Download - UOG Journal Club: July 2014
UOG Journal Club: July 2014Interobserver agreement in describing adnexal masses using the International Ovarian Tumor
Analysis simple rules in a real-time setting and using three-dimensional ultrasound volumes and digital clips
B. RUIZ DE GAUNA, P. SANCHEZ, L. PINEDA, J. UTRILLA-LAYNA, L. JUEZ, J. L. ALCÁZAR
Volume 44, Issue 1, Date: July 2014, pages 95-99
Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven observers
L. ZANNONI, L. SAVELLI, L. JOKUBKIENE, A. DI LEGGE, G. CONDOUS, A. C. TESTA, P. SLADKEVICIUS, L. VALENTIN
Volume 44, Issue 1, Date: July 2014, pages 100-108
Journal Club slides prepared by Dr Tommaso Bignardi(UOG Editor for Trainees)
Background
• In 2008, the International Ovarian Tumor Analysis (IOTA) group proposed the so-called ‘simple rules’ for ultrasound classification of adnexal masses.
• Simple rules can be applied in about 75–80% of all adnexal masses.
• When the mass is classified as benign or malignant using simple rules, the diagnostic performance is good, even when applied by examiners with differing levels of expertise.
Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional
ultrasound volumes and digital clipsRuis de Gauna et al., UOG 2014
Objective: To estimate agreement between expert and non-expert
examiners using the IOTA simple rules for classifying adnexal masses on real-time ultrasound and using three-dimensional
(3D) ultrasound volumes and digital clips
Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional
ultrasound volumes and digital clipsRuis de Gauna et al., UOG 2014
Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional
ultrasound volumes and digital clipsRuis de Gauna et al., UOG 2014
N=42 women with adnexal mass were evaluated by transvaginal power Doppler ultrasound
Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional
ultrasound volumes and digital clipsRuis de Gauna et al., UOG 2014
Examination by non-expert (trainee)
Expert examination
Videoclips &Static 3D volumes acquired by expert
Assessed by four trainees with different levels of training
Methods
• Examiners were blinded to each other.
• Examiners had to classify the mass as benign, malignant or inconclusive using real-time ultrasound evaluation, short video clips (about 15–20s) and by manipulating 3D volumes.
• Agreement in classifying the mass as benign, inconclusive or malignant was estimated by weighted kappa index and percentage of agreement.
• Agreement for each ultrasound feature was estimated by the standard kappa index.
Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional
ultrasound volumes and digital clipsRuis de Gauna et al., UOG 2014
Methods
Features for malignant tumor Features for benign tumor
M1 Irregular solid tumor B1 Unilocular tumor
M2 Presence of ascites B2 Presence of solid components where solid component largest
diameter is <7mm
M3 At least four papillary projections B3 Presence of acoustic shadows
M4 Irregular multilocular solid tumor with largest diameter ≥100mm
B4 Smooth multilocular tumor with largest diameter <100mm
M5 Very strong blood flow (color score 4) B5 No blood flow (color score 1)
Timmerman et al, UOG 2008
Simple rules for classifying adnexal masses proposed by the International Ovarian Tumor Analysis (IOTA) group
Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional
ultrasound volumes and digital clipsRuis de Gauna et al., UOG 2014
Trainee
Benign Inconclusive Malignant Total
Expert examiner
Benign 17 0 0 17
Inconclusive 4 2 2 8
Malignant 1 2 14 17
Total 22 4 16 42
Weighted kappa = 0.76 (95% CI, 0.61–0.90); percentage of agreement=78.6% (33 of 42)
Results: Agreement between expert examiner and trainee on real-time ultrasound examination with regard to classifying
adnexal masses as benign, malignant or inconclusive
Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional
ultrasound volumes and digital clipsRuis de Gauna et al., UOG 2014
FeaturePrevalence %
(n/n)*Kappa index
(95% CI)Percentage agreement
(n/n)
B1: Unilocular tumor 17.8 (15/84) 0.89 (0.75 to 1.0) 95.2 (40/42)
B2: Presence of solid components where solid component largest diameter is <7mm
4.8 (4/84) 0.64 (0.19 to 1.0) 95.2 (40/42)
B3: Presence of acoustic shadows 4.8 (4/84) 0.36 (−0.20 to 0.92)
92.9 (39/42)
B4: Smooth multilocular tumor with largest diameter <100mm 8.3 (7/84) 0.91 (0.73 to 1.0) 97.6 (41/42)
B5: No blood flow (color score 1) 19.0 (16/84) 0.53 (0.25 to 0.82) 80.9 (34/42)
M1: Irregular solid tumor 14.3 (12/84) 0.58 (0.27 to 0.88) 85.7 (36/42)
M2: Presence of ascites 10.7 (9/84) 0.42 (0.03 to 0.80) 85.7 (36/42)
M3: At least four papillary projections 4.8 (4/84) 0.64 (0.19 to 1.0) 95.2 (40/42)
M4: Irregular multilocular solid tumor with largest diameter ≥100mm 8.3 (7/84) 0.81 (0.55 to 1.0) 95.2 (40/42)
M5: Very strong blood flow (color score 4) 28.6 (24/84) 0.63 (0.40 to 0.85) 80.9 (34/42)
*Number of times the feature is observed by any observer/total number of observations.
Results: Agreement between expert examiner and trainee on real-time ultrasound for each ultrasound feature
3rd year trainee 2nd year trainee 1st year trainee
4th year trainee WK=0.56 (0.33–0.80) [76.2%]
WK=0.58 (0.35–0.81) [76.2%]
WK=0.51 (0.27–0.73) [71.4%]
3rd year trainee - WK=0.51 (0.26–0.74) [72.2%]
WK=0.46 (0.21–0.71) [71.4%]
2nd year trainee - - WK=0.45 (0.20-0.71) [71.4%]
Data are given as weighted kappa (WK), (95% CI), [percentage of agreement]
Results: Agreement between trainees with regard to classifying adnexal masses as benign, malignant or unclassifiable when analyzing video clips
and three-dimensional volumes
Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional
ultrasound volumes and digital clipsRuis de Gauna et al., UOG 2014
• Good agreement with regard to classifying adnexal masses as benign, malignant or inconclusive using the IOTA simple rules between an expert and a less experienced examiner.
• Moderate agreement using a video clip plus a 3D volume for trainees with different degrees of training.
• ‘Presence of acoustic shadows’ and ‘Presence of ascites’ showed the worst agreement between expert and trainee on real-time imaging.
Results
Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional
ultrasound volumes and digital clipsRuis de Gauna et al., UOG 2014
• First study assessing interobserver agreement with regard to describing adnexal masses using the IOTA simple rules during real-time ultrasound.
Strengths of the study
Weaknesses
• The series is small and comprises of a selected population.
• No sample size estimation with wide CIs for the kappa index.
• Analysis does not include inherent sources of variability between different acquisitions.
Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional
ultrasound volumes and digital clipsRuis de Gauna et al., UOG 2014
Background
• Even when using standardized terms and definitions, ultrasound examiners may evaluate the features of an adnexal mass differently.
• Consequently, the risk of malignancy calculated by mathematical models may vary both within and between examiners.
• Problems that ultrasound examiners may encounter when using gray-scale IOTA terminology for adnexal masses have not been evaluated in any published study.
Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven
observersZannoni et al., UOG 2014
Objective: To estimate intraobserver repeatability and interobserver agreement in assessing the presence of papillary projections in adnexal masses
and in classifying adnexal masses using the IOTA terminology for ultrasound examiners with different levels of experience
Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven
observersZannoni et al., UOG 2014
• Digital clips (2-8 clips per adnexal mass) with gray-scale and color/power Doppler information of 83 adnexal masses were acquired.
• Evaluated independently by 4 experienced and 3 less experienced ultrasound observers.
• Four evaluations: 2 before and 2 after a consensus meeting.
• Variables analyzed were tumor type and presence of papillary projections.
Methods
Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven
observersZannoni et al., UOG 2014
Parameter Agreement (%) Fleiss kappa Cohen’s kappa
First evaluation before meeting
All observers (n=7) 76.0 (60.2–85.5) 0.695 (0.687–0.703) 0.690 (0.500–0.810)
Experienced observers (n=4) 78.7 (72.3–81.9) 0.735 (0.726–0.744) 0.727 (0.653–0.762)
Less experienced observers (n=3) 70.6 (67.5–74.7) 0.621 (0.611–0.632) 0.617 (0.576–0.672)
First evaluation after meeting
All observers (n=7) 75.4 (66.3–85.5) 0.682 (0.675–0.689) 0.682 (0.554–0.812)
Experienced observers (n=4) 73.5 (69.9–75.9) 0.660 (0.651–0.669) 0.661 (0.613–0.695)
Less experienced observers (n=3) 79.1 (75.9–84.3) 0.726 (0.717–0.734) 0.727 (0.685–0.797)
Data are given as mean (range) for pairwise agreement and pairwise Cohen’s kappa, and point estimate (95% CI) for Fleiss kappa
Results: Interobserver agreement with regard to tumor type among seven observers before and after a consensus meeting
Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven
observersZannoni et al., UOG 2014
Parameter Agreement (%) Fleiss kappa Cohen’s kappa
First evaluation before meeting
All observers (n=7) 86.6 (75.9–92.3) 0.536 (0.522–0.550) 0.520 (0.148–0.747)
Experienced observers (n=4) 86.1 (80.7–89.2) 0.579 (0.562–0.597) 0.578 (0.381–0.643)
Less experienced observers (n=3) 87.1 (84.3–89.2) 0.441(0.419–0.463) 0.434 (0.348–0.577)
First evaluation after meeting
All observers (n=7) 82.7 (68.7–92.8) 0.487 (0.474–0.500) 0.480 (0.194–0.787)
Experienced observers (n=4) 83.9 (75.9–92.7) 0.588 (0.572–0.603) 0.602 (0.490–0.787)
Less experienced observers (n=3) 84.7 (80.7–90.4) 0.397 (0.373–0.421) 0.414 (0.318–0.470)
Data are given as mean (range) for pairwise agreement and pairwise Cohen’s kappa, and point estimate (95% CI) for Fleiss kappa
Results: Interobserver agreement with regard to papillary projections among seven observers before and after a consensus meeting
Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven
observersZannoni et al., UOG 2014
Before meeting After meeting
Observer Agreement (%)
Kappa (95%CI) Agreement (%)
Kappa (95%CI)
Most experienced observers
A 87.9 0.84 (0.75–0.93) 91.6 0.89 (0.81–0.97)
B 86.7 0.83 (0.74–0.92) 90.3 0.86 (0.78–0.94)
C 89.1 0.86 (0.78–0.94) 86.7 0.83 (0.73–0.92)
D 86.7 0.83 (0.74–0.92) 86.8 0.83 (0.74–0.92)
Less experienced observers
E 92.7 0.90 (0.82–0.98) 92.7 0.90 (0.86–0.94)
F 95.1 0.94 (0.88–1.0) 90.3 0.87 (0.79–0.95)
G 95.2 0.94 (0.88–1.0) 86.8 0.83 (0.74–0.92)
Results: Intraobserver repeatability in 83 cases of adnexal mass before and after a consensus meeting when classifying tumors
Before meeting After meeting
Observer Agreement (%)
Kappa (95%CI) Agreement (%)
Kappa (95%CI)
Most experienced observers
A 91.6 0.72 (0.52–0.92) 96.4 0.89 (0.77–1.0)
B 94.0 0.83 (0.69–0.97) 95.1 0.90 (0.80–1.0)
C 92.8 0.77 (0.59–0.95) 88.0 0.67 (0.48–0.86)
D 94.0 0.80 (0.63–0.97) 92.8 0.81 (0.67–0.95)
Less experienced observers
E 98.8 0.88 (0.66–1.0) 96.4 0.75 (0.47–1.0)
F 91.6 0.76 (0.59–0.93) 90.4 0.76 (0.60–0.92)
G 100.0 1.0 (1.0–1.0) 90.3 0.64 (0.41–0.87)
Results: Intraobserver repeatability in 83 cases of adnexal mass with regard to the presence of papillary projections before and after a consensus meeting
Results:
• There was uncertainty about how to define a solid component and a papillary projection, but consensus was reached at the consensus meeting.
• Interobserver agreement was good for tumor type.
• Interobserver agreement for papillary projection was moderate but very variable between observer pairs.
Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven
observersZannoni et al., UOG 2014
• First study assessing problems that ultrasound examiners may encounter in describing gray-scale ultrasound images of adnexal masses using IOTA terminology.
Strengths of the study
Weaknesses• No sample size estimation with wide CIs for the kappa index.
• Results might not be applicable to live ultrasound examinations.
• Results are generalizable only to adnexal masses similar to those in the study, and to observers with similar experience.
Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven
observersZannoni et al., UOG 2014
• The IOTA terms ‘papillary projection’ and ‘solid component’ might need a more precise definition.
• Practical advice is needed on how to discriminate between a solid component and ovarian tissue, a solid component and densely packed septa or small cysts, a solid component and a tangentially cut septum, etc.
• The consensus meeting did not change inter- or intraobserver agreement.
Discussion
Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven
observersZannoni et al., UOG 2014
Discussion points
• What is the clinical evidence for the utility of ultrasound classification of tumors to dictate subsequent clinical management?
• How are ovarian tumor types classified by ultrasound in your unit?
• Are videos/3D volume acquisition a source of significant variability in the evaluation of ovarian masses?
• How should we define a papillary projection or a solid component of an adnexal mass using IOTA terminology?
• How does the known reproducibility of the test affect estimation of risk of malignancy using IOTA or simple rules