The Risk of Ovarian Malignancy Algorithm (ROMA) was developed for assessing the risk of ovarian cancer (OC) among women with a pelvic mass. The risk score is calculated based on serum human epididymis protein 4 (HE4) and cancer antigen 125 (CA125) concentrations and is then compared with cutoff values for high risk of OC. ROMA may be used by primary physicians or gynecologists who are not oncology specialists to decide whether to refer a patient to an appropriate center for surgery [1^,2^,3].

The cutoff ROMA score was determined according to the expected specificity or sensitivity needed by gynecologists in clinical studies wherein EOC and borderline ovarian tumor (BOT) or low malignant potential (LMP) tumor were considered as target conditions [2^,4]. It is provided by the manufacturer of the HE4 and CA125 assays. However, different cutoffs have been used in ROMA evaluation studies [5^,6], and attempts have been made to determine thresholds specific to the relevant study population [7^,8^,9^,10^,11]. Although there may be no significant difference in ROMA performance using suggested cutoff values and adjusted values [11], laboratories have been advised to establish optimized cutoff values for ROMA as well as HE4 according to the relevant study population [5].

Studies examining ROMA's diagnostic accuracy in detecting OC often calculate sensitivity at a set specificity of ≥75.0% and compare it with that of other markers such as Risk of Malignancy Index and Copenhagen Index [2^,3^,4^,11^,12^,13^,14^,15^,16^,17]. This is because a specificity of 75.0% has been generally assumed to be the clinically acceptable minimum [3]. The optimal ROMA cutoff score and clinical usefulness of the test were determined with a sensitivity >80.0% at a specificity of 75.0% [2^,4^,18].

The reported sensitivity and specificity of ROMA are variable [4^,8^,9^,12^,15^,16^,17^,19]. This variation has been attributed to differences in geographic backgrounds, the number of study subjects, and the analytic systems used for HE4 and CA125 measurements among studies [20]. Differences in the disease spectrum of the study populations and prevalence of target condition between the studies may also contribute to their inconsistent and variable findings [21]. Moore et al. [2] developed ROMA from a population with an OC prevalence of about 30.0% (151/503). Since then, the OC prevalence reported in ROMA diagnostic accuracy studies generally ranges between 15.7% and 45.8% [8^,9^,15^,16]. However, ROMA can be tested on a population with a lower EOC prevalence in real situations.

To our knowledge, no studies have established whether the clinically acceptable minimal sensitivity (>80.0%) and intended specificity (≥75.0%) can be obtained using the suggested cutoff of 7.4%/25.3% for pre/postmenopausal women in routine clinical practice, even in populations with a lower OC prevalence than that reported in diagnostic accuracy studies. We attempted to fill this gap. If the clinical expectation was not achieved with the suggested cutoff, we determined the adjusted cutoff value required.

The distributions of serum HE4, CA125, and ROMA scores are shown in Fig. 2. The patient distributions between the conditions determined by reference standard and ROMA risk groups, according to the manufacturer's suggested cutoff, are shown in Table 3. ROMA diagnostic accuracy parameters (i.e., sensitivity, specificity, PPV, and NPV) were calculated using the suggested cutoff of 7.4%/25.3%; with specificity cutoffs of 75.0%, 80.0%, 90.0%, or 95.0%; or with a sensitivity cutoff of 95.0% (Table 4 and Supplemental Data Table S1).

Considering BOT and EOC as the target condition, the cutoff set to a specificity of 75.0% was 4.65%/13.71%, with a sensitivity of 81.8%. The adjusted cutoff for BOT and EOC was also lower than the suggested cut off, 7.40%/25.30% (Table 4). Considering EOC as the target condition, the cutoff (pre/postmenopausal) at a specificity of 75.0% was 4.78%/14.35%, which was lower than the suggested cutoff, 7.40%/25.30%; when the adjusted cutoff was applied, the sensitivity was 88.9% (Supplemental Data Table S1).

We aimed to determine whether ROMA could reach the clinically acceptable minimal sensitivity and the intended specificity in clinical practice for a patient population with a lower prevalence of OC when using the suggested cutoff, and to determine an adjusted cutoff to achieve the expected specificity or sensitivity.

When BOT and EOC were considered as target conditions, the sensitivity of ROMA using the suggested cutoff was much lower than the clinically acceptable minimum of 80.0%. The specificity remained considerably higher than the 75.0% specificity intended. Using the adjusted cutoff, the sensitivity reached 80.0%. Considering only EOC as the target condition, the sensitivity of ROMA was slightly lower than or close to the clinically acceptable minimal value even with the suggested cutoff. The specificity was much higher than the intended specificity. After adjusting the cutoff value, the sensitivity of ROMA would be higher than the acceptable minimum.

Several ROMA studies have used the ARCHITECT HE4 and CA 125 assay. Chan et al. [8] used the suggested cutoff of 7.4%/25.3% for a population of Asian women with an EOC prevalence of 15.7% and reported that the sensitivity, specificity, PPV, and NPV of ROMA were 89.2%, 87.3%, 58.6%, and 97.6%, respectively. Their EOC prevalence was three to four times higher than that reported in this study (4.1%). Although sensitivity and PPV were higher than those in the current study, specificity and NPV were similar. Karlsen et al. [15] used the suggested cutoff of 7.4%/25.3% to identify EOC and reported a sensitivity of 93.9%/97.5% (pre/postmenopausal women) and specificity of 52.6%/57.0% (pre/postmenopausal women). Specificity was lower than intended, while sensitivity met the clinically acceptable minimum. Conversely, specificity was 76.5% using a cutoff set to a sensitivity of 94.4%, while sensitivity was 94.8% using a cutoff set to a specificity of 75.0%. In both cases, the acceptable minimal specificity of 75.0% and sensitivity of 80.0% were achieved. This was also observed in the current study. Although the EOC prevalence in the study by Karlsen et al. [15] was 21.0%, five times higher than that reported in this study (4.1%), the sensitivity (88.9% with a cutoff set to a specificity of 75.0%) and specificity (73.6% with a cutoff set to a sensitivity of 95.0%) in the current study were slightly lower, but similar to their reported sensitivity (94.8% with a cutoff set to a specificity of 75.0%) and specificity (76.5% with a cutoff set to a sensitivity of 94.4%) [15].

Winarto et al. [9] reported that the sensitivity, specificity, PPV, and NPV of ROMA with the suggested cutoffs of 7.4%/25.3% were 94.0%, 42.6%, 57.3%, and 89.7%, respectively, for identifying EOC in a study population with a prevalence of 39.1%. When identifying BOT as well as EOC in the same study, the parameters were 91.0%, 42.6%, 63.5%, and 81.3% while the prevalence was 52.3% in the population [9]. In the study by Moore et al. [3], the sensitivity, PPV, and NPV of ROMA at a set specificity of 75.0% were 94.3%, 59.8%, and 97.1%, respectively, for EOC with prevalence of 28.0%. When targeting BOT and EOC in the same study, the parameters were 89.0%, 62.3%, and 93.6% while the prevalence was 32.0% [3]. The findings in the two studies [3^,9] were consistent with our results that the sensitivity (63.6% vs 77.8%) and NPV (96.9% vs 99.0%) of targeting BOT and EOC were lower while PPV was higher (35.6% vs 23.7%) than those of targeting EOC only for the same study population. Their higher sensitivity and PPV with lower specificity and NPV, relative to those of the current study, may be due to their much higher prevalence.

According to Sandri et al. [16], when applying the suggested cutoff of 7.4%/25.3% for their population with an EOC prevalence of 45.8% (160/349), the sensitivity and the specificity of ROMA were 89.3% and 81.7%, respectively. The sensitivity of ROMA at a set specificity of 75.0%, 80.0%, and 90.0% was 91.2%, 91.3%, and 84.4%, respectively [16]. Both the suggested cutoff and the cutoff set to a specificity reached the acceptable minimal values for sensitivity (>80.0%) and specificity (≥75.0%). Kim et al. [19] determined the thresholds with a specificity of 95% in a study involving subjects with the same ethnic background as that in this study. The reported EOC and BOT prevalence was 47.1% (72/153). Using the resultant cutoffs of 7.6%/10.9%, sensitivity was 87.5%. In the current study, the cutoffs set to a specificity of 95% were 10.78%/36.44%, with a low sensitivity of 57.6%, when EOC and BOT were considered as target conditions. Across three [8^,9^,15] of four studies using the ARCHITECT HE4 and CA 125 assays and the suggested cutoff value for identifying EOC [8^,9^,15^,16], lower EOC prevalence was accompanied by lower sensitivity and higher specificity, even though the range of prevalence in those three studies was limited and far from that of the current study (4.1%). Lower sensitivity and higher specificity were observed with the suggested cutoff in the population with a lower OC prevalence, consistent with our findings. Therefore, we infer that the sensitivity and specificity of ROMA in clinical practice may be different from that reported in previous studies, particularly in hospitals with a lower OC prevalence.

One further factor requires consideration. If ROMA diagnostic accuracy studies are conducted at relatively large hospitals receiving referrals of patients with suspected OC, the OC prevalence reported in those studies is likely to be high [3^,8^,9^,15^,16^,19]. However, the OC prevalence among patients encountered in clinical practice in the referring hospitals may be lower than that reported in previous ROMA diagnostic accuracy studies. Moreover, the patient spectrum relevant to the target condition may differ between referral-receiving hospitals and referring hospitals. Our institute is a middle-to-large-sized secondary hospital; accordingly, it accepts referrals from primary clinics and also refers patients across all departments to tertiary hospitals. Therefore, the OC prevalence was likely lower than that reported in other ROMA diagnostic accuracy studies [3^,8^,9^,15^,16^,19].

This study has some limitations. Because the OC prevalence was low, subgroup analyses according to histological type and stage were not feasible. In addition, as clinical follow-up with imaging studies was used as an alternative for patients with no pathological examination, two types of reference standards were used. The adjusted cutoff values derived from the current study should be applied to other patient groups in future studies for validation.

To our knowledge, this is the first study examining the diagnostic accuracy of ROMA in clinical practice at a middle-to-large sized institute where the OC prevalence was as low as that of a hospital referring patients with pelvic mass to gynecologic oncologists at a tertiary care center. As the study population was a consecutive series of patients, this study reflects the real clinical situation of a referring hospital.

In conclusion, when the cutoff suggested by the manufacturer was used in clinical practice for a patient population with a lower OC prevalence, the intended specificity of 75.0% was clearly exceeded. However, the minimal acceptable sensitivity was not achieved when both BOT and EOC were considered as the target condition. Conversely, the sensitivity almost reached the acceptable minimum of 80.0% when EOC was the only target condition. When the cutoff value was modified to set the specificity to 75.0%, the sensitivity increased to a level above the acceptable minimum, both when EOC and BOT and only EOC were used as the target condition. Finally, the expected diagnostic accuracy of ROMA may not be observed in all hospitals, depending on the characteristics of the patient population and the prevalence of the target condition.

Flow chart of patients.

Abbreviations: ROMA, risk of ovarian malignancy algorithm; EOC, epithelial ovarian cancer; BOT, borderline ovarian tumor.

Serum concentrations of (A) HE4, (B) CA125, and (C) ROMA for each disease group and menopausal status (N=443). The cutoffs are shown as horizontal lines.

Abbreviations: HE4, human epididymis protein 4; CA125, cancer antigen 125; ROMA, risk of ovarian malignancy algorithm; BOT, borderline ovarian tumor; EOC, epithelial ovarian cancer; Non-EOC, non-epithelial ovarian cancer.