Article

Original Article

Ann Lab Med 2023; 43(6): 539-553

Published online November 1, 2023 https://doi.org/10.3343/alm.2023.43.6.539

Copyright © Korean Society for Laboratory Medicine.

Neutrophil Gelatinase-Associated Lipocalin Cutoff Value Selection and Acute Kidney Injury Classification System Determine Phenotype Allocation and Associated Outcomes

Annemarie Albert, M.D.1 , Sebastian Radtke, Cand.Med.2,* , Louisa Blume, Cand.Med.2,* , Rinaldo Bellomo, M.D.3,4 , Michael Haase, M.D.5,6,7 , Philipp Stieger, M.D., M.M.E.2 , Ulrich Paul Hinkel, M.D.8 , Rüdiger C. Braun-Dullaeus, M.D.2, and Christian Albert, M.D.2,8

1Department of Nephrology and Endocrinology, Ernst von Bergmann Hospital, Potsdam, Germany; 2University Clinic for Cardiology and Angiology, Otto-von-Guericke University Magdeburg, Germany; 3Department of Intensive Care, The Austin Hospital, Melbourne, Australia; 4Centre for Integrated Critical Care, The University of Melbourne, Melbourne, Australia; 5Medical Faculty, Otto-von-Guericke University Magdeburg, Germany; 6Diamedikum, Potsdam, Germany; 7Department of Nephrology and Hypertension, Hannover Medical School, Hannover, Germany; 8Department of Nephrology, Central Clinic Bad Berka, Bad Berka, Germany

Correspondence to: Christian Albert, M.D.
University Clinic for Cardiology and Angiology, Otto-von-Guericke University Magdeburg, Leipziger Straße 44, 39120 Magdeburg, Germany
Tel: +49-391-6713203
Fax: +49-391-6713202
E-mail: Christian.Albert@med.ovgu.de

* These authors equally contributed to this study.

Received: January 9, 2023; Revised: February 19, 2023; Accepted: June 15, 2023

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Background: We explored the extent to which neutrophil gelatinase-associated lipocalin (NGAL) cutoff value selection and the acute kidney injury (AKI) classification system determine clinical AKI-phenotype allocation and associated outcomes.
Methods: Cutoff values from ROC curves of data from two independent prospective cardiac surgery study cohorts (Magdeburg and Berlin, Germany) were used to predict Kidney Disease: Improving Global Outcome (KDIGO)- or Risk, Injury, Failure, Loss of kidney function, End-stage (RIFLE)-defined AKI. Statistical methodologies (maximum Youden index, lowest distance to [0, 1] in ROC space, sensitivity≈specificity) and cutoff values from two NGAL meta-analyses were evaluated. Associated risks of adverse outcomes (acute dialysis initiation and in-hospital mortality) were compared.
Results: NGAL cutoff concentrations calculated from ROC curves to predict AKI varied according to the statistical methodology and AKI classification system (10.6–159.1 and 16.85–149.3 ng/mL in the Magdeburg and Berlin cohorts, respectively). Proportions of attributed subclinical AKI ranged 2%–33.0% and 10.1%–33.1% in the Magdeburg and Berlin cohorts, respectively. The difference in calculated risk for adverse outcomes (fraction of odds ratios for AKI-phenotype group differences) varied considerably when changing the cutoff concentration within the RIFLE or KDIGO classification (up to 18.33- and 16.11-times risk difference, respectively) and was even greater when comparing cutoff methodologies between RIFLE and KDIGO classifications (up to 25.7-times risk difference).
Conclusions: NGAL positivity adds prognostic information regardless of RIFLE or KDIGO classification or cutoff selection methodology. The risk of adverse events depends on the methodology of cutoff selection and AKI classification system.

Keywords: Acute kidney injury, Cardiac surgery, Neutrophil gelatinase-associated lipocalin, Subclinical AKI, AKI phenotypes, Cutoff, Risk prediction, Risk assessment, Dichotomization, ROC

There is a persistent need for early identification of patients at risk of cardiac surgery-associated acute kidney injury (AKI) because of the increased need for kidney replacement therapy (KRT) and increased risk of in-hospital mortality [1].

Despite advances in the conceptual understanding of the pathophysiology and epidemiology of cardiac surgery-associated AKI [2], consensus definitions and staging of AKI remain based on relative changes in serum creatinine (sCr) concentration, a decline in urine output (UO) [3], and the provision of KRT [4].

Several prospective observational studies have assessed whether the use of kidney tubular damage biomarkers such as urine neutrophil gelatinase-associated lipocalin (NGAL) improves early kidney risk assessment when measured in addition to sCr concentration [5-8]. The performance of such biomarkers as predictors of subsequent changes in sCr concentrations is poor, which may be influenced by various confounders such as the choice of the consensus definition of AKI, clinical setting, or type of sample used to measure NGAL (urine or plasma) [9].

Patients testing positive for kidney tubular damage biomarkers may not fulfill the traditional AKI criteria and may still have worse outcomes compared to those with negative biomarker findings. This recognition led to the proposal that new AKI diagnostic criteria should equally include both glomerular function markers (sCr) and kidney damage biomarkers (e.g., NGAL or other injury biomarkers) based on thresholds; however, there is still no consensus of the appropriate thresholds for such assessments [10]. Multiple statistical methodologies have been suggested for determining the “optimal” cutoff values for such tasks [11]. Therefore, using multiple methodologies, studies have determined and reported study-specific optimal NGAL cutoff concentrations to define NGAL positivity or negativity [5-7, 12-15].

We hypothesized that differences in the methodology of NGAL cutoff value determination and the system chosen for AKI classification influence patients’ clinical AKI-phenotype group allocation and the associated risk of adverse events according to the proposed AKI matrix system [10, 16]. Using data from two independent prospective cardiac surgery study cohorts [17, 18], we aimed to explore the extent to which such changes in methodologies and classification systems affect the calculated risk of adverse outcome [15, 16]. We hypothesized that patient outcomes and the magnitude of the attributed risk of adverse events between clinical AKI-phenotype groups vary according to the methodology used for NGAL cutoff selection and the choice of classification system for sCr/UO-based AKI.

Study design

We used the existing data of 200 patients at an increased risk of AKI from the Berlin study cohort of a randomized, multicenter study on cardiac surgery (NCT00672334) performed at the German Heart Center Berlin between May 2008 and January 2012 [17]. Urine samples were obtained upon patients’ admission to the intensive care unit (ICU) and were immediately centrifuged at 5,000 rpm (~ 3,075 g) and stored at −80°C. Following completion of patient enrollment, NGAL concentrations in urine samples were measured using badge analysis as described below.

The second study cohort included 103 consecutively enrolled adult patients undergoing open-heart surgery at the University Clinic Magdeburg, Germany, which aimed to explore postoperative clinical kidney risk assessment using urine sampled at ICU admission for each patient and NGAL measured individually within 60 minutes according to routine laboratory diagnostics [18].

In both cohorts, NGAL measurements were performed in the central laboratory at the Institute of Clinical Chemistry and Pathobiochemistry of the University Clinic Magdeburg, using a standardized clinical platform assay for urinary NGAL (ng/mL; ARCHITECT, Abbott Diagnostics, Abbott Park, IL, USA), which is a Conformité Européenne (CE)-certified biomarker for the diagnosis of AKI in Germany. The CE certification mark indicates conformity with the health, safety, and environmental protection standards for products sold within the European Economic Area. The measurement interval of the ARCHITECT urine NGAL assay is 10.0–1,500.0 ng/mL, with an imprecision of ≤10% total CV. When using an automated dilution procedure, the assay can report values of up to 6,000.0 ng/mL. Further data on assay performance and handling requirements used in this study are described elsewhere [19]. The sCr concentrations were measured via routine laboratory diagnostics using an enzymatic method standardized by isotope dilution-mass spectroscopy on a Cobas 8000 modular analyzer (Roche Diagnostics, Mannheim, Germany). Complete sCr and urinary NGAL data were available for 199/200 and 100/103 patients of the Berlin and Magdeburg cohorts, respectively. Patients with missing data were excluded from analysis.

Laboratory investigators were blinded to the sample sources and clinical outcomes. In the Berlin study cohort, Medicine Ethics Committee, Charité University, Berlin, Germany approved the study (ZS EK 11 654/07), and written informed consent was obtained from each patient [17]. In the study performed at the University Clinic Magdeburg, Germany, the local institutional Review Board categorized the study as an audit of current clinical practice and approved prospective data collection and use for the purpose of this study. Patients received written study information, but the need for written consent was waived by the ethics committee (Ethics Committee, University of Magdeburg, Case 49/13) [18]. Full study details have been described previously [17, 18].

Definition of sCr/UO-based AKI

sCr/UO-based AKI was defined according to the Renal risk, Injury, Failure, Loss of kidney function, End-stage (RIFLE) [3] or Kidney Disease: Improving Global Outcomes (KDIGO) classification system [4], considering increases in postoperative sCr concentrations from baseline (preoperative stage) and the decline in UO criteria in both cohorts. Both classification systems define AKI as an increase in sCr concentration by a factor of 1.5-times from baseline or a decline in UO to less than 0.5 mL/kg/hr for up to 6–12 hours. In contrast to RIFLE, the KDIGO classification additionally considers an increase of ≥0.3 mg/dL (≥26.5 µmol/L) within 48 hours as AKI stage 1.

Methodology of NGAL cutoff selection

We used five methods to derive candidate NGAL cutoff values for clinical AKI phenotypes. In both datasets, the area under the ROC curve (AUC) value was calculated separately for the dichotomous outcome measure sCr/UO-based AKI according to RIFLE and KDIGO classification criteria fulfilling at least RIFLE-risk or KDIGO stage 1, respectively. We chose candidate cutoff values based on study-patient ROC curve analysis [11, 20], where each point on the ROC curve corresponds to a potential cutoff value and is associated with a sensitivity and specificity value pair. Threshold selection represents a compromise between sensitivity and specificity (Fig. 1) [21]. Previously, to derive the NGAL and sCr/UO-based AKI matrix, there has been no preference of favoring either high sensitivity or specificity over the compromise for an optimal balance of both indices [22]. Similarly, for the present derivation of cutoff values, no weighting of sensitivity and specificity was applied.

Figure 1. Derivation of a 2×2 contingency table according to a chosen NGAL cutoff value in an example distribution of patients with and without sCr/UO-based AKI. (A) Example distribution of sCr/UO-based AKI and AKI-free patients according to NGAL concentration on the X-axis. When these distributions overlap in their NGAL concentrations, type 1 (FP) and type 2 (FN) errors are introduced. The ROC space was then defined by the FPR (1–specificity) and the TPR. An ROC curve for a dichotomous outcome measure is generated by a cohort’s finite set of 2×2 cell matrices or contingency tables, where each represents a trade-off between specificity and sensitivity pairs. Decreasing the cutoff value would result in fewer FNs but consecutively increase the number of FPs (B). Increasing the cutoff value would result in fewer FPs but consecutively more FNs (C). The proportions of attributed clinical AKI phenotypes corresponding to the figure can be derived depending on the chosen NGAL cutoff values in the 2×2 table [16, 33].
Abbreviations: AKI, acute kidney injury; FN, false negative; FP, false positive; FPR, false positive rate; NGAL, neutrophil gelatinase-associated lipocalin; RIFLE, Risk, Injury, Failure, Loss, and End-stage; sCr, serum creatinine; TPR, true positive rate; UO, urine output.

To determine an optimal cutoff value to differentiate between patients with the condition (sCr/UO-AKI) and those free of the condition (AKI-free), various methods have been proposed aiming to narrow the gap to the point in the ROC space that mathematically maximizes the AUC to “perfect test prediction”; that is, the point at coordinates (0, 1) in the upper-left corner of the ROC space.

The first method is based on previous studies [5, 13] that selected the maximum Youden index [23] as the cutoff value, which maximizes the sum of sensitivity and specificity by maximizing the distance from the diagonal chance line (y=x) connecting (0, 0) to (1, 1) in the ROC space. The chance line itself approximates an AUC value of 0.5. The Youden index for a given point on the ROC curve is calculated as follows:

Youden index (J) = Sensitivity + Specificity-1

The maximum Youden index is the point on the ROC curve farthest from the chance line [11, 24].

The second method is one that balances sensitivity and specificity in assessing when the test sensitivity is equivalent to the test specificity. This point on the curve is located on the line connecting the upper-left corner (0, 1) to the lower-right corner (1, 0) of the ROC space. Hypothetically, at the point where sensitivity equals specificity, the product of these two indices (sensitivity×specificity) is at its maximum [11]. In this study, we chose the point closest to the equivalence of sensitivity and specificity, referred to as sensitivity≈specificity.

The third method minimizes the distance of a candidate cutoff value to the point on the curve closest to perfection (0, 1). This cutoff value should correspond to the optimal cutoff value chosen from all potential points available on the ROC curve [24].

Mathematically, searching for the shortest radius originating from (0, 1), the square of the distance from each data point on the ROC curve to (0, 1) is first calculated as:

d2=(1-TPF)+ FPF2=(1-Sensitivity)2 + (1-Specificty)2

where TPF is the true-positive fraction and FPF is the false-positive fraction [24]. The distance D to (0, 1) of any given point on the ROC curve is then calculated as follows:

D=d2.

Finally, two cutoff values were adopted from two meta-analyses investigating the ability of urinary NGAL to predict sCr/UO-based AKI [9, 25].

Derivation of the AKI phenotype matrix

Patients were allocated to the clinical AKI phenotype matrix groups derived from each candidate cutoff value of 2×2 contingency table cells from the ROC curve with regard to meeting or not meeting the criteria for any stage of sCr/UO-based AKI, separately for RIFLE and KDIGO (functional impairment, dichotomization).

NGAL positivity (+), indicating the presence of structural or tubular damage, or NGAL negativity (–) was determined according to whether the urinary NGAL concentration measured at ICU admission after cardiac surgery was above or equal (≥) or lower (<) than each candidate cutoff value, respectively.

In summary, the following four different combinations defining the patient groups could be distinguished considering the RIFLE/KDIGO and NGAL statuses individually for each candidate cutoff value (Fig. 2): NGAL(–)/sCr/UO(–), NGAL(+)/sCr/UO(–), NGAL(–)/sCr/UO(+), and NGAL(+)/sCr/UO(+).

Figure 2. Matrix of attributed clinical AKI phenotypes derived from the NGAL test-based 2×2 contingency table. (A) Attributed clinical AKI phenotypes plotted in a 2×2 contingency table or “matrix.” Functional impairment is attributed to increased sCr concentrations or reduced UO and defined by positive RIFLE or KDIGO criteria. Structural damage is attributed to increased NGAL concentrations above a candidate threshold concentration. Three scenarios of potential functional and structural kidney impairment can be distinguished. (B) Corresponding test classification matrix: NGAL(–)/sCr)/UO(–), TN, no kidney impairment; NGAL(+)/sCr/UO(–), FP, subclinical AKI (AKI stage 1 S); NGAL(–)/sCr/UO(+), FN, hemodynamic AKI, volume depletion, diminished kidney functional reserve (AKI stage 1–3A); NGAL(+)/sCr/UO(+), TP, AKI with functional and structural impairment (AKI stage 1–3B).
Abbreviations: AKI, acute kidney injury; FN, false negative; FP, false positive; KDIGO, Kidney Disease Improving Global Outcomes; NGAL, neutrophil gelatinase-associated lipocalin; sCr, serum creatinine; TN, true negative; TP, true positive; UO, urine output.

Outcome measures

The predefined outcome measures were acute KRT initiation, in-hospital mortality, and the combination of KRT or in-hospital mortality.

Statistical analysis

The odds ratio (OR) with 95% confidence interval (CI) was calculated to assess the associated difference in the risk of developing an outcome measure between the patient groups derived from the corresponding AKI phenotype matrix. When there were groups with zeros (N=0), we introduced a corrective factor of 0.5 for all cells. The two groups of zeros were not corrected or compared with respect to systematic differences. We then calculated the magnitude of risk disparity between the methodologies used for cutoff value selection with the following formula:

Magnitude of risk disparity =ORmethodology aORmethodology b

Since the OR is a relative measure of risk, a value of 1.0 would correspond to the assumption of no difference in the magnitude of risk of developing an outcome of interest between the cutoff selection methodologies “a” and “b.” SPSS version 28.0 (IBM Corp., Armonk, NY, USA), R Environment for Statistical Computing [26], and Microsoft Excel 365 version 16.43 (Redmond, WA, USA) were used for statistical analyses.

In the Magdeburg cohort, 14 (14.0%) patients had AKI according to the RIFLE classification and 23 (23.0%) had AKI according to the KDIGO classification, whereas in the Berlin cohort, 24 (12.1%) patients had AKI according to the RIFLE classification and 59 (29.6%) patients had AKI according to the KDIGO classification. The detailed perioperative patient characteristics of both cohorts are presented in Supplemental Data Table S1 and in Supplemental Data Fig. S1. The ROC curves and associated AUC values of NGAL in predicting RIFLE- and KDIGO-defined AKI in both cohorts are presented in Supplemental Data Fig. S2. All derived cutoff values and their associated statistical metrics are listed in Table 1.

Table 1 . Risk assessment metrics according to various methods used to calculate the urine NGAL cutoff concentration for AKI (all stages) defined by the RIFLE and KDIGO classification systems in the Magdeburg and Berlin cohorts

MethodologyNGAL cutoff*, ng/mLSensitivitySpecificityDYouden JDOR (95% CI)LR+ (95% CI)LR– (95% CI)PPV (95% CI)NPV (95% CI)AKI prevalence, %
Magdeburg cohort

RIFLE AKI classification used as dependent outcome measure
Sensitivity ≈ specificity10.60.7140.7090.4080.4236.10 (1.75–21.28)2.46 (1.54–3.92)0.40 (0.17–0.93)0.29 (0.14–0.44)0.94 (0.88–1.00)14.0
Lowest distance D to (0,1)11.750.7140.7330.3910.4476.85 (1.95–24.00)2.67 (1.65–4.32)0.39 (0.17–0.90)0.30 (0.15–0.46)0.94 (0.88–1.00)14.0
Maximum Youden index45.550.5000.9530.5020.45320.50 (4.80–87.46)10.75 (3.61–32.00)0.52 (0.31–0.89)0.64 (0.35–0.92)0.92 (0.87–0.98)14.0
Albert, et al. [9]§66.30.3570.9650.6440.32215.37 (3.14–75.23)10.24 (2.75–38.14)0.67 (0.45–0.99)0.63 (0.29–0.96)0.90 (0.84–0.96)14.0
Haase, et al. [25]||159.10.2140.9650.7870.1797.55 (1.35–42.12)6.14 (1.37–27.46)0.81 (0.62–1.07)0.50 (0.10–0.90)0.88 (0.82–0.95)14.0
KDIGO AKI classification used as dependent outcome measure
Sensitivity ≈ specificity8.00.5650.5710.6110.1361.73 (0.68–4.44)1.32 (0.85–2.05)0.76 (0.46–1.26)0.28 (0.15–0.41)0.81 (0.71–0.92)23.0
Lowest distance D to (0,1)9.50.5650.6880.5350.2532.87 (1.10–7.46)1.81 (1.11–2.96)0.63 (0.39–1.03)0.35 (0.20–0.51)0.84 (0.75–0.93)23.0
Maximum Youden index45.550.3480.9610.6530.30913.16 (3.12–55.43)8.93 (2.58–30.93)0.68 (0.50–0.92)0.73 (0.46–0.99)0.83 (0.75–0.91)23.0
Albert, et al. [9]§66.30.2610.9740.7390.23513.24 (2.46–71.35)10.04 (2.17–46.43)0.76 (0.59–0.97)0.75 (0.45–1.05)0.82 (0.74–0.89)23.0
Haase, et al. [25]||159.10.1740.9740.8260.1487.89 (1.34–46.37)6.70 (1.31–34.25)0.85 (0.70–1.03)0.67 (0.29–1.04)0.80 (0.72–0.88)23.0

Berlin cohort

RIFLE AKI classification used as dependent outcome measure
Sensitivity ≈ specificity16.850.6250.6230.5320.2482.75 (1.14–6.64)1.66 (1.15–2.38)0.60 (0.35–1.02)0.19 (0.10–0.27)0.92 (0.88–0.97)12.1
Lowest distance to (1,0)50.00.6250.7710.4390.3965.28 (2.15–12.93)2.60 (1.73–3.91)0.49 (0.29–0.83)0.26 (0.15–0.38)0.94 (0.90–0.98)12.1
Maximum Youden index50.00.6250.7710.4390.3965.28 (2.15–12.93)2.60 (1.73–3.91)0.49 (0.29–0.83)0.26 (0.15–0.38)0.94 (0.90–0.98)12.1
Albert, et al. [9]§79.30.5420.8110.4950.3535.09 (2.09–12.36)2.87 (1.78–4.64)0.56 (0.36–0.88)0.28 (0.15–0.41)0.93 (0.89–0.97)12.1
Haase, et al. [25]||149.250.4580.8460.5630.3044.64 (1.88–11.43)2.97 (1.70–5.18)0.64 (0.44–0.93)0.29 (0.15–0.43)0.92 (0.88–0.96)12.1
KDIGO AKI classification used as dependent outcome measure
Sensitivity ≈ specificity14.250.6270.6290.5260.2562.85 (1.52–5.34)1.69 (1.26–2.26)0.59 (0.42–0.85)0.42 (0.31–0.52)0.80 (0.73–0.87)29.7
Lowest distance to (1,0)14.750.6270.6360.5210.2632.93 (1.56–5.51)1.72 (1.28–2.31)0.59 (0.41–0.84)0.42 (0.32–0.52)0.80 (0.73–0.88)29.7
Maximum Youden index36.50.5080.7790.5390.2873.64 (1.90–6.95)2.30 (1.54–3.42)0.63 (0.48–0.83)0.49 (0.37–0.62)0.79 (0.72–0.86)29.7
Albert, et al. [9]§79.30.3560.8210.6680.1772.54 (1.28–5.05)1.99 (1.22–3.27)0.78 (0.64–0.96)0.46 (0.31–0.60)0.75 (0.68–0.82)29.7
Haase, et al. [25]||149.250.3050.8570.7100.1622.63 (1.27–5.46)2.14 (1.22–3.74)0.81 (0.68–0.97)0.47 (0.31–0.63)0.75 (0.68–0.81)29.7

*Considered a positive result if greater than or equal to the cutoff; D2=(1–TPF)2+FPF2=(1–sensitivity)2+(1–specificity)2; Youden index (J) defined as TPF–FPF=sensitivity+specificity−1; §We chose the value closest to that suggested by Albert, et al. [9] (81 ng/mL); ||We chose the value closest to that suggested by Haase, et al. [25] (150 ng/mL).

Abbreviations: AKI, acute kidney injury; KDIGO, Kidney Disease Improving Global Outcomes; NGAL, neutrophil gelatinase-associated lipocalin; NPV, negative predictive value; PPV, positive predictive value; RIFLE, Risk, Injury, Failure, Loss, End-stage; TPF, true positive fraction; FPF, false positive fraction.



The cutoff concentrations calculated from the corresponding ROC curves differed according to both the methodology used and the choice of RIFLE or KDIGO classification, ranging from 10.6 to 159.1 ng/mL in the Magdeburg cohort and from 16.85 to 149.25 ng/mL in the Berlin cohort. In both cohorts, the findings obtained when using the sensitivity≈specificity and lowest distance methods were similar. In the Berlin cohort, using the RIFLE classification, the lowest distance method yielded the same cutoff value as the maximum Youden index. Lower cutoff values subsided with distinctly higher sensitivity for sCr/UO-based AKI (RIFLE and KDIGO), as opposed to specificity improvements for higher cutoff values.

The fraction of potential subclinical AKI (AKI stage 1 S) shifted from 2% to 33% and from 10.1% to 26.1% for the KDIGO classification in the Magdeburg and Berlin cohorts, respectively (Table 2). With increasing cutoff values, more patients were eventually allocated to the NGAL-negative group.

Table 2 . Patients’ allocation and outcome according to NGAL/RIFLE and NGAL/KDIGO groups, N (%)

Cutoff methodologysCr/UO-AKINGAL cutoff* (ng/mL)NGAL(–)/ sCr/UO(–)NGAL(+)/ sCr/UO(–)NGAL(–)/ sCr/UO(+)§NGAL(+)/ sCr/UO(+)ll
Allocation

Magdeburg cohort
Sensitivity ≈ specificityRIFLE10.661 (61.0)25 (25.0)4 (4.0)10 (10.0)
Lowest distance D to (0,1)RIFLE11.7563 (63.0)23 (23.0)4 (4.0)10 (10.0)
Maximum Youden indexRIFLE45.5582 (82.0)4 (4.0)7 (7.0)7 (7.0)
Albert, et al. [9]RIFLE66.383 (83.0)3 (3.0)9 (9.0)5 (5.0)
Haase, et al. [25]**RIFLE159.183 (83.0)3 (3.0)11 (11.0)3 (3.0)
Sensitivity ≈ specificityKDIGO8.044 (44.0)33 (33.0)10 (10.0)13 (13.0)
Lowest distance D to (0,1)KDIGO9.553 (53.0)24 (24.0)10 (10.0)13 (13.0)
Maximum Youden indexKDIGO45.5574 (74.0)3 (3.0)15 (15.0)8 (8.0)
Albert, et al. [9]KDIGO66.375 (75.0)2 (2.0)17 (17.0)6 (6.0)
Haase, et al. [25]**KDIGO159.175 (75.0)2 (2.0)19 (19.0)4 (4.0)
Berlin cohort
Sensitivity ≈ specificityRIFLE16.85109 (54.8)66 (33.2)9 (4.5)15 (7.5)
Lowest distance D to (0,1)RIFLE50.0133 (66.8)42 (21.1)9 (4.5)15 (7.5)
Maximum Youden indexRIFLE50.0133 (66.8)42 (21.1)9 (4.5)15 (7.5)
Albert, et al. [9]RIFLE79.3142 (71.4)33 (16.6)11 (5.5)13 (6.5)
Haase, et al. [25]**RIFLE149.25148 (74.4)27 (13.6)13 (6.5)11 (5.5)
Sensitivity ≈ specificityKDIGO14.2588 (44.2)52 (26.1)22 (11.1)37 (18.6)
Lowest distance D to (0,1)KDIGO14.7589 (44.7)51 (25.6)22 (11.1)37 (18.6)
Maximum Youden indexKDIGO36.5109 (54.8)31 (15.6)29 (14.6)30 (15.1)
Albert, et al. [9]KDIGO79.3115 (57.8)25 (12.6)38 (19.1)21 (10.6)
Haase, et al. [25]**KDIGO149.25120 (60.3)20 (10.1)41 (20.6)18 (9.1)

KRT initiation

Magdeburg cohort
Sensitivity ≈ specificityRIFLE10.60 (0.0)0 (0.0)0 (0.0)6 (60.0)
Lowest distance D to (0,1)RIFLE11.750 (0.0)0 (0.0)0 (0.0)6 (60.0)
Maximum Youden indexRIFLE45.550 (0.0)0 (0.0)2 (28.6)4 (57.1)
Albert, et al. [9]RIFLE66.30 (0.0)0 (0.0)3 (33.3)3 (60.0)
Haase, et al. [25] **RIFLE159.10 (0.0)0 (0.0)5 (45.5)1 (33.3)
Sensitivity ≈ specificityKDIGO8.00 (0.0)0 (0.0)0 (0.0)6 (46.2)
Lowest distance D to (0,1)KDIGO9.50 (0.0)0 (0.0)0 (0.0)6 (46.2)
Maximum Youden indexKDIGO45.550 (0.0)0 (0.0)2 (13.3)4 (50.0)
Albert, et al. [9]KDIGO66.30 (0.0)0 (0.0)3 (17.6)3 (50.0)
Haase, et al. [25]**KDIGO159.10 (0.0)0 (0.0)5 (26.3)1 (25.0)
Berlin cohort
Sensitivity ≈ specificityRIFLE16.852 (1.8)3 (4.5)1 (11.1)7 (46.7)
Lowest distance D to (0,1)RIFLE50.02 (1.5)3 (7.1)1 (11.1)7 (46.7)
Maximum Youden indexRIFLE50.02 (1.5)3 (7.1)1 (11.1)7 (46.7)
Albert, et al. [9]RIFLE79.34 (2.8)1 (3.0)1 (9.1)7 (53.8)
Haase, et al. [25]**RIFLE149.254 (2.7)1 (3.7)2 (15.4)6 (54.5)
Sensitivity ≈ specificityKDIGO14.250 (0.0)0 (0.0)3 (13.6)10 (27.0)
Lowest distance D to (0,1)KDIGO14.750 (0.0)0 (0.0)3 (13.6)10 (27.0)
Maximum Youden indexKDIGO36.50 (0.0)0 (0.0)3 (10.3)10 (33.3)
Albert, et al. [9]KDIGO79.30 (0.0)0 (0.0)5 (13.2)8 (38.1)
Haase, et al. [25]**KDIGO149.250 (0.0)0 (0.0)6 (14.6)7 (38.9)

In-hospital mortality

Magdeburg cohort
Sensitivity ≈ specificityRIFLE10.60 (0.0)0 (0.0)0 (0.0)3 (30.0)
Lowest distance D to (0,1)RIFLE11.750 (0.0)0 (0.0)0 (0.0)3 (30.0)
Maximum Youden indexRIFLE45.550 (0.0)0 (0.0)1 (14.3)2 (28.6)
Albert, et al. [9]RIFLE66.30 (0.0)0 (0.0)2 (22.2)1 (20.0)
Haase, et al. [25]**RIFLE159.10 (0.0)0 (0.0)2 (18.2)1 (33.3)
Sensitivity ≈ specificityKDIGO8.00 (0.0)0 (0.0)0 (0.0)3 (23.1)
Lowest distance D to (0,1)KDIGO9.50 (0.0)0 (0.0)0 (0.0)3 (23.1)
Maximum Youden indexKDIGO45.550 (0.0)0 (0.0)1 (6.7)2 (25.0)
Albert, et al. [9]KDIGO66.30 (0.0)0 (0.0)2 (11.8)1 (16.7)
Haase, et al. [25]**KDIGO159.10 (0.0)0 (0.0)2 (10.5)1 (25.0)
Berlin cohort
Sensitivity ≈ specificityRIFLE16.851 (0.9)4 (6.1)2 (22.2)6 (40.0)
Lowest distance D to (0,1)RIFLE50.01 (0.8)4 (9.5)2 (22.2)6 (40.0)
Maximum Youden indexRIFLE50.01 (0.8)4 (9.5)2 (22.2)6 (40.0)
Albert, et al. [9]RIFLE79.33 (2.1)2 (6.1)3 (27.3)5 (38.5)
Haase, et al. [25]**RIFLE149.253 (2.0)2 (7.4)3 (23.1)5 (45.5)
Sensitivity ≈ specificityKDIGO14.251 (1.1)2 (3.8)2 (9.1)8 (21.6)
Lowest distance D to (0,1)KDIGO14.751 (1.1)2 (3.9)2 (9.1)8 (21.6)
Maximum Youden indexKDIGO36.51 (0.9)2 (6.5)2 (6.9)8 (26.7)
Albert, et al. [9]KDIGO79.32 (1.7)1 (4.0)4 (10.5)6 (28.6)
Haase, et al. [25]**KDIGO149.252 (1.7)1 (5.0)4 (9.8)6 (33.3)

KRT initiation or in-hospital mortality

Magdeburg cohort
Sensitivity ≈ specificityRIFLE10.60 (0.0)0 (0.0)0 (0.0)6 (60.0)
Lowest distance D to (0,1)RIFLE11.750 (0.0)0 (0.0)0 (0.0)6 (60.0)
Maximum Youden indexRIFLE45.550 (0.0)0 (0.0)2 (28.6)4 (57.1)
Albert, et al. [9]RIFLE66.30 (0.0)0 (0.0)3 (33.3)3 (60.0)
Haase, et al. [25]**RIFLE159.10 (0.0)0 (0.0)5 (45.5)1 (33.3)
Sensitivity ≈ specificityKDIGO8.00 (0.0)0 (0.0)0 (0.0)6 (46.2)
Lowest distance D to (0,1)KDIGO9.50 (0.0)0 (0.0)0 (0.0)6 (46.2)
Maximum Youden indexKDIGO45.550 (0.0)0 (0.0)2 (13.3)4 (50.0)
Albert, et al. [9]KDIGO66.30 (0.0)0 (0.0)3 (17.6)3 (50.0)
Haase, et al. [25]**KDIGO159.10 (0.0)0 (0.0)5 (26.3)1 (25.0)
Berlin cohort
Sensitivity ≈ specificityRIFLE16.853 (2.8)5 (7.6)2 (22.2)11 (73.3)
Lowest distance D to (0,1)RIFLE50.03 (2.3)5 (11.9)2 (22.2)11 (73.3)
Maximum Youden indexRIFLE50.03 (2.3)5 (11.9)2 (22.2)11 (73.3)
Albert, et al. [9]RIFLE79.36 (4.2)2 (6.1)3 (27.3)10 (76.9)
Haase, et al. [25] **RIFLE149.256 (4.1)2 (7.4)4 (30.8)9 (81.8)
Sensitivity ≈ specificityKDIGO14.251 (1.1)2 (3.8)4 (18.2)14 (37.8)
Lowest distance D to (0,1)KDIGO14.751 (1.1)2 (3.9)4 (18.2)14 (37.8)
Maximum Youden indexKDIGO36.51 (0.9)2 (6.5)4 (13.8)14 (46.7)
Albert, et al. [9]KDIGO79.32 (1.7)1 (4.0)7 (18.4)11 (52.4)
Haase, et al. [25]**KDIGO149.252 (1.7)1 (5.0)8 (19.5)10 (55.6)

*A positive test result is determined if greater than or equal to the cutoff value; No damage or functional impairment [10, 16]; Subclinical AKI (stage 1 S); kidney tubular injury indicated by positive NGAL, no functional impairment detected [10, 16]; §Hemodynamic AKI (stage 1–3A); no tubular injury detected, functional impairment present [10, 16]; llAKI stage 1–3B; tubular injury and functional impairment present [10, 16]; We chose the value closest to that suggested by Albert, et al. [9] (81 ng/mL); **We chose the value closest to that suggested by Haase, et al. [25] (150 ng/mL).

Abbreviations: KDIGO, Kidney Disease Improving Global Outcomes; KRT, kidney replacement therapy; NGAL, neutrophil gelatinase-associated lipocalin; RIFLE, Risk, Injury, Failure, Loss, and End-Stage Kidney Disease; sCr, serum creatinine; UO, urine output.



When the highest cutoff values were chosen, more patients with adverse events (i.e., those who were more likely to have higher NGAL concentrations) were reassigned to the NGAL-negative group, which resulted in groups with NGAL-negative status having more events than those with NGAL-positive status (OR <1.0; Table 2 and Table 3, Fig. 1C).

Table 3 . Odds ratios for the risk of the outcome measures between the NGAL/RIFLE and NGAL/KDIGO groups

Cutoff methodologysCr/UO-AKINGAL(+)/sCr/UO(–)* vs. referenceNGAL(–)/sCr(+) vs. referenceNGAL(+)/sCr/UO(+)§ vs. referenceNGAL(+)/sCr(+) vs. NGAL(–)/sCr/UO(+)
KRT initiation

Magdeburg cohort
Sensitivity ≈ specificityRIFLENCNC177.67 (8.57–3,682.96)13.00 (0.55–306.22)
Lowest distance D to (0,1)RIFLENCNC183.44 (8.85–3,801.49)13.00 (0.55–306.22)
Maximum Youden indexRIFLENC75.00 (3.19–1,762.00)212.14 (9.45–4,763.31)2.83 (0.37–21.89)
Albert, et al. [9]RIFLENC89.92 (4.18–1,934.88)233.80 (9.35–5,847.08)2.60 (0.33–20.79)
Haase, et al. [25]**RIFLENC141.31 (7.01–2,847.09)100.20 (3.22–3,121.90)0.71 (0.07–7.22)
Sensitivity ≈ specificityKDIGONCNC77.13 (3.92–1,516.64)18.20 (0.88–374.91)
Lowest distance D to (0,1)KDIGONCNC92.73 (4.73–1,818.94)18.20 (0.88–374.91)
Maximum Youden indexKDIGONC27.59 (1.25–607.18)149.00 (6.90–3,218.77)5.40 (0.83–35.33)
Albert, et al. [9]KDIGONC36.45 (1.79–743.94)151.00 (6.46–3,527.54)4.14 (0.63–27.32)
Haase, et al. [25]**KDIGONC57.28 (3.00–1,093.38)64.71 (2.21–1,891.82)1.13 (0.13–9.70)
Berlin cohort
Sensitivity ≈ specificityRIFLE2.55 (0.41–15.66)6.69 (0.55–81.94)46.81 (8.32–263.50)7.00 (0.69–70.75)
Lowest distance D to (0,1)RIFLE5.04 (0.81–31.24)8.19 (0.67–100.18)57.31 (10.20–321.99)7.00 (0.69–70.75)
Maximum Youden indexRIFLE5.04 (0.81–31.24)8.19 (0.67–100.18)57.31 (10.20–321.99)7.00 (0.69–70.75)
Albert, et al. [9]RIFLE1.08 (0.12–9.97)3.45 (0.35–33.85)40.25 (9.20–176.03)11.67 (1.14–119.55)
Haase, et al. [25]**RIFLE1.38 (0.15–12.89)6.55 (1.08–39.79)43.20 (9.19–203.09)6.60 (0.97–44.93)
Sensitivity ≈ specificityKDIGONC31.77 (1.58–640.36)67.58 (3.83–1,190.96)2.13 (0.56–8.14)
Lowest distance D to (0,1)KDIGONC32.13 (1.59–647.54)68.35 (3.88–1,204.32)2.13 (0.56–8.14)
Maximum Youden indexKDIGONC28.92 (1.45–577.18)112.17 (6.32–1,990.32)3.88 (1.02–14.81)
Albert, et al. [9]KDIGONC37.93 (2.04–703.57)145.44 (7.94–2,662.96)3.84 (1.10–13.32)
Haase, et al. [25]**KDIGONC44.13 (2.43–802.56)157.17 (8.43–2,931.68)3.56 (1.03–12.35)

In-hospital mortality

Magdeburg cohort
Sensitivity ≈ specificityRIFLENCNC57.40 (2.69–1,222.58)4.20 (0.17–101.54)
Lowest distance D to (0,1)RIFLENCNC59.27 (2.78–1,261.93)4.20 (0.17–101.54)
Maximum Youden indexRIFLENC38.08 (1.41–1,031.09)60.00 (2.40–1,497.18)1.58 (0.14–17.41)
Albert, et al. [9]RIFLENC55.67 (2.44–1,269.49)37.11 (1.09–1,259.86)0.67 (0.05–8.95)
Haase, et al. [25]**RIFLENC43.95 (1.96–985.05)66.80 (1.79–2,495.20)1.52 (0.10–22.74)
Sensitivity ≈ specificityKDIGONCNC29.67 (1.42–619.31)7.00 (0.32–152.96)
Lowest distance D to (0,1)KDIGONCNC35.67 (1.71–742.79)7.00 (0.32–152.96)
Maximum Youden indexKDIGONC15.41 (0.60–397.42)57.31 (2.48–1,324.62)3.72 (0.40–34.43)
Albert, et al. [9]KDIGONC24.35 (1.11–532.69)41.18 (1.50–1,134.17)1.69 (0.18–15.98)
Haase, et al. [25]**KDIGONC21.57 (0.99–469.65)64.71 (2.21–1,891.82)3.00 (0.29–30.74)
Berlin cohort
Sensitivity ≈ specificityRIFLE6.97 (0.76–63.74)30.86 (2.48–383.23)72.00 (7.79–665.30)2.33 (0.36–15.30)
Lowest distance D to (0,1)RIFLE13.89 (1.51–128.04)37.71 (3.04–467.80)88.00 (9.54–811.96)2.33 (0.36–15.30)
Maximum Youden indexRIFLE13.89 (1.51–128.04)37.71 (3.04–467.80)88.00 (9.54–811.96)2.33 (0.36–15.30)
Albert, et al. [9]RIFLE2.99 (0.48–18.65)17.38 (3.01–100.17)28.96 (5.85–143.28)1.67 (0.29–9.45)
Haase, et al. [25]**RIFLE3.87 (0.61–24.32)14.50 (2.59–81.29)40.28 (7.75–209.29)2.78 (0.48–16.03)
Sensitivity ≈ specificityKDIGO3.48 (0.31–39.35)8.70 (0.75–100.74)24.00 (2.88–200.14)2.76 (0.53–14.38)
Lowest distance D to (0,1)KDIGO3.59 (0.32–40.63)8.80 (0.76–101.89)24.28 (2.91–202.41)2.76 (0.53–14.38)
Maximum Youden indexKDIGO7.45 (0.65–85.05)8.00 (0.70–91.53)39.27 (4.67–330.09)4.91 (0.94–25.53)
Albert, et al. [9]KDIGO2.35 (0.21–27.03)6.65 (1.17–37.88)22.60 (4.18–122.30)3.40 (0.84–13.84)
Haase, et al. [25]**KDIGO3.11 (0.27–35.95)6.38 (1.12–36.24)29.50 (5.35–162.61)4.63 (1.11–19.19)

KRT initiation or in-hospital mortality

Magdeburg cohort
Sensitivity ≈ specificityRIFLENCNC177.67 (8.57–3,682.96)13.00 (0.55–306.22)
Lowest distance D to (0,1)RIFLENCNC183.44 (8.85–3,801.49)13.00 (0.55–306.22)
Maximum Youden indexRIFLENC75.00 (3.19–1,762.00)212.14 (9.45–4,763.31)2.83 (0.37–21.89)
Albert, et al. [9]RIFLENC89.92 (4.18–1,934.88)233.80 (9.35–5,847.08)2.60 (0.33–20.79)
Haase, et al. [25]**RIFLENC141.31 (7.01–2,847.09)100.20 (3.22–3,121.90)0.71 (0.07–7.22)
Sensitivity ≈ specificityKDIGONCNC77.13 (3.92–1,516.64)18.20 (0.88–374.91)
Lowest distance D to (0,1)KDIGONCNC92.73 (4.73–1,818.94)18.20 (0.88–374.91)
Maximum Youden indexKDIGONC27.59 (1.25–607.18)149.00 (6.90–3,218.77)5.40 (0.83–35.33)
Albert, et al. [9]KDIGONC36.45 (1.79–743.94)151.00 (6.46–3,527.54)4.14 (0.63–27.32)
Haase, et al. [25]**KDIGONC57.28 (3.00–1,093.38)64.71 (2.21–1,891.82)1.13 (0.13–9.70)
Berlin cohort
Sensitivity ≈ specificityRIFLE2.90 (0.67–12.54)10.10 (1.44–70.66)97.17 (19.22–491.30)9.63 (1.38–67.25)
Lowest distance D to (0,1)RIFLE5.86 (1.34–25.65)12.38 (1.77–86.51)119.17 (23.62–601.29)9.63 (1.38–67.25)
Maximum Youden indexRIFLE5.86 (1.34–25.65)12.38 (1.77–86.51)119.17 (23.62–601.29)9.63 (1.38–67.25)
Albert, et al. [9]RIFLE1.46 (0.28–7.59)8.50 (1.79–40.39)75.56 (16.40–348.05)8.89 (1.40–56.58)
Haase, et al. [25]**RIFLE1.89 (0.36–9.92)10.52 (2.51–44.10)106.50 (18.76–604.57)10.13 (1.47–69.94)
Sensitivity ≈ specificityKDIGO3.48 (0.31–39.35)19.33 (2.04–183.31)52.96 (6.61–423.97)2.74 (0.77–9.76)
Lowest distance D to (0,1)KDIGO3.59 (0.32–40.63)19.56 (2.06–185.40)53.57 (6.69–428.79)2.74 (0.77–9.76)
Maximum Youden indexKDIGO7.45 (0.65–85.05)17.28 (1.85–161.36)94.50 (11.62–768.34)5.47 (1.53–19.59)
Albert, et al. [9]KDIGO2.35 (0.21–27.03)12.76 (2.52–64.53)62.15 (12.06–320.26)4.87 (1.49–15.95)
Haase, et al. [25]**KDIGO3.11 (0.27–35.95)14.30 (2.90–70.62)73.75 (13.76–395.16)5.16 (1.54–17.27)

*NGAL(+)/sCr/UO(–), Subclinical AKI (stage 1 S); kidney tubular injury indicated by positive NGAL; The reference group was NGAL(–)/sCr/UO(–), No damage or functional impairment [10, 16]. NGAL(–)/sCr(+), Hemodynamic AKI (stage 1–3A); §NGAL(+)/sCr(+), AKI stage 1–3B; tubular injury and functional impairment present [10, 16]; We chose the value closest to that suggested by Albert, et al. [9] (81 ng/mL); **We chose the value closest to that suggested by Haase, et al. [25] (150 ng/mL).

NC: When there were groups with zeros, N=0, we introduced a corrective factor of 0.5 to all cells. The two groups of zeros were not corrected for and were not compared with respect to systematic differences.

Abbreviations: KDIGO, Kidney Disease Improving Global Outcomes; KRT, kidney replacement therapy; NC, not calculable; NGAL, neutrophil gelatinase-associated lipocalin; RIFLE, Risk, Injury, Failure, Loss, and End-stage; sCR, serum creatinine; UO, urine output; OR, odds ratio.



The proportion of patients with kidney impairment additionally identified by urinary NGAL only in relation to the proportion of patients diagnosed as having AKI by conventional sCr/UO-based criteria differed by up to 8.33–16.49 times and 2.46–2.6 times between the lowest and highest candidate cutoffs for the RIFLE and KDIGO classifications in the Berlin and Magdeburg cohorts, respectively (Supplemental Data Fig. S3 and S4). The distribution of patients according to the derived cutoff values allocated to the NGAL/sCr/UO groups and their outcomes are shown in Table 2. The ORs illustrating the differences in the risk of developing adverse events between these groups are provided in Table 3.

Finally, the magnitude of the difference in the calculated risk (fraction of ORs) for adverse events between the group distributions varied considerably when the cutoff concentration was changed within the RIFLE or KDIGO classification (up to 18.33- and 16.11-times risk difference, respectively). This variation in the calculated risk was even more prominent when comparing the various cutoff methodologies between the RIFLE and KDIGO classification systems, reaching up to a 12.79-times difference in the Berlin cohort and up to a 25.67-times difference in the Magdeburg cohort for KRT initiation. As an example, such distinct changes in calculated risk resulted from an approximate doubling of the NGAL cutoff value from 79.3 to 149.3 ng/mL and the change of the NGAL cutoff value from approximately 10 to 160 ng/mL, respectively (Fig. 3, Supplemental Data Fig. S5A–D).

Figure 3. Heat maps illustrating the magnitude of difference of calculated risk. Heat map examples show the magnitude of difference of calculated risk derived by the division of ORs from Table 3 for the development of the outcome measures of KRT initiation or in-hospital mortality (A, B) or in-hospital mortality (C). A factor of 1.0 indicates no difference between the methodologies. (A) For attributed subclinical phenotype (AKI stage 1 S) NGAL(+)/sCr/UO(–) vs. NGAL(–)/sCr/UO(–) reference groups in the Berlin cohort. As an example, the calculated risk (OR) between these groups varied by a factor of 4 when choosing 50 vs. 79.3 ng/mL as a cutoff concentration for NGAL using the RIFLE classification. (B) NGAL(+)/sCr/UO(+) vs. NGAL(–)/sCr(+) groups in the Magdeburg cohort. For example, the calculated risk (OR) between these groups varied by a factor of 16.1 when choosing 9.5 vs. 159.1 ng/mL as a cutoff concentration for NGAL using the KDIGO classification. The highest variation observed was a 25.67-times difference of the calculated risk between using KDIGO and 8.0 ng/mL as the NGAL cutoff vs. RIFLE and 159.1 ng/mL as the NGAL cutoff concentration. (C) Such variation was also present comparing attributed hemodynamic/pre-kidney AKI phenotypes NGAL(–)/sCr(+) vs. NGAL(–)/sCr(–) as reference for the outcome in-hospital mortality in the Berlin cohort. The calculated risk (OR) between these groups varied by a factor of 2.17 when choosing RIFLE and 50.0 vs. 79.3 ng/mL or a 5.91-times difference when choosing KDIGO with 149.25 ng/mL over RIFLE and 50.0 ng/mL as the cutoff concentration for NGAL.
Abbreviations: AKI, acute kidney injury; KRT, kidney replacement therapy; KDIGO, Kidney Disease: Improving Global Outcome classification; NGAL, neutrophil gelatinase-associated lipocalin; OR, odds ratio; RIFLE, Risk, Injury, Failure, Loss of kidney function, End-stage; sCr, serum creatinine; UO, urine output.

Using two independent cardiac surgery cohorts, we assessed the extent to which the methodology for NGAL cutoff value selection and AKI classification system affected patients’ clinical AKI-phenotype allocation and the associated risk of KRT initiation or in-hospital mortality.

We found an increased risk of KRT initiation and in-hospital mortality for patients testing NGAL-positive. This increased risk for adverse events found between clinical phenotype groups varied considerably in magnitude (up to a 25.7-times difference) according to the underlying sCr/UO classification system of AKI and the NGAL cutoff selection methodology applied.

At ICU admission, NGAL positivity carried more prognostic information over NGAL-negative findings, regardless of whether the kidney function acutely declined (positive RIFLE/KDIGO criteria), confirming the findings of a previous study [5]. However, multiple methods of determining attributed “optimal” cutoff values to define biomarker positivity have been suggested [11]. These thresholds were derived from ROC curves to predict sCr and UO-based AKI [5, 12, 13].

Previous studies on NGAL predicting adverse outcomes in various AKI settings selected certain cutoffs based on published values, manufacturer recommendations, or calculated cutoff values from the study population [5-7, 12-15]. Considering a variety of confounders such as the AKI classification system and the collection timing of samples for NGAL testing, a recent meta-analysis derived summary ROC curves and NGAL cutoff values for the prediction of sCr-based AKI [9], showing high variability in the data among the included studies, which may be closely related to the present findings on the risk disparity for adverse events.

Considering that higher NGAL concentrations likely reflect more severe kidney injury than lower concentrations, the associated risk of adverse events and specificity will steadily increase as the measured concentration increases [27]. Although cardiac surgery-associated AKI is common, severe adverse complications such as KRT initiation or mortality are less frequent [28, 29], which seems to be consistent with the lower frequency of high NGAL concentrations in the present cohorts.

Given the small samples investigated in these cohorts with relatively few adverse events, our data indicate that there may be study-specific “sweet-spot thresholds” where the risk prediction metrics (Table 1) will be the highest, resulting in favorable allocation of patients according to such specific NGAL cutoff values that more distinctively differentiate groups of relative risk of adverse events [30]. It is therefore reasonable that the applied cutoff value selection methods may identify the same cutoff values as “optimal” in a small cohort but identify different cutoff values as “optimal” in larger cohorts [24]. This may also explain the high level of heterogeneity of previously reported cutoff values [9]. Even a small shift in the cutoff concentration (ng/mL) may ultimately lead to a potentially meaningful clinical difference in outcome distribution and consecutive risk disparity, as the respective cutoff 2×2 table is subject to change (Figs. 1 and 2) [31].

Extending conventional AKI classification criteria to a clinically applicable AKI phenotype matrix-based system representing different pathophysiological phenotypes of AKI (Fig. 2) is desirable [32]. Characterization of clinical AKI phenotypes in such a matrix-based system is directly dependent on the cutoff value chosen for dichotomization [33]. In such a matrix-based framework, a biomarker may be of additional prognostic value only if it clearly differentiates the incremental risk of adverse events within the derived matrix-based groups rather than specifically showing good diagnostic performance for sCr/UO-based AKI according to a high AUC value [34].

We found that the statistical methodology for NGAL cutoff value selection may be a relevant factor potentially leading to inconsistent findings in patients’ phenotype allocation and in the calculation of the attributed risk of adverse events among different clinical phenotypes of AKI. We consider that the magnitude of such confounding factors will likely depend on the variability across patient cohorts and that calculations of risk differences may be affected by cohort sample size variations. Complementarily, in the aforementioned recent meta-analysis on NGAL cutoff values [9], we demonstrated that the underlying meta-analysis approach may also lead to considerable differences in findings for discriminatory estimators such as the AUC or derived cutoff values [35].

Many caveats remain to be resolved to more accurately define AKI phenotypes [32]. NGAL was previously associated with mortality in cardiogenic shock [36], but concentrations and discriminative ability may also be influenced by systemic conditions such as sepsis [37] or variations in urine flow [38], potentially confounding the findings. We did not correct the outcome data for the presence of septic conditions as influencing factors since supplemental data were not recorded [17, 18]; however, including C-reactive protein, a surrogate parameter for sepsis, did not improve the risk assessment of AKI phenotypes in a previous analysis of this cohort [5]. In the present study, the risk parameters and cutoff values were derived from and were dependent on the investigated cohorts; therefore, they may not be generalizable. Nevertheless, the internal validity of our findings is strengthened by the similarities in risk variability within two independent, well-defined cardiac surgery cohorts with a consequent timing of biomarker sampling. We do not preclude the conclusion that other biomarkers for AKI risk prediction may also be affected by changing biomarker threshold selections [39].

Changing cutoff concentrations or threshold methodology may also change the previous pathophysiological narrative and associated understanding of AKI phenotypes such as subclinical AKI [16, 33]. We suggest considering potential confounders [9] and emphasize that great care should be taken when establishing a cutoff concentration for clinical use. The variability in risk in our findings highlights the complementary prognostic relevance of kidney functional parameters and kidney injury biomarkers and the necessity to interpret both in conjunction with the clinical context. Further studies are needed to assess the potential clinical relevance of the cutoff value selection methodology [40] and to outline the associated risk disparity for the clinical phenotypes of AKI.

In summary, using two independent cardiac surgery cohorts, we confirmed our hypothesis that the magnitude of the attributed risk of adverse events varies according to the methodology used for NGAL cutoff selection and the classification system of sCr/UO-based AKI. Further studies are needed to address alternative methods and evaluate the advantages and limitations of different approaches, in addition to those based on the ROC curve and dichotomization.

Albert C, Albert A, Bellomo R, and Braun-Dullaeus RC contributed to the conception and design of the study. Haase M and Albert C acquired patient data. Albert C and Albert A performed formal analysis, interpretation and illustration of the data, and wrote the first draft of the manuscript. Albert C supervised the study. All authors participated in drafting and/or revising the paper and provided intellectual content of critical importance to the work, read and approved the final manuscript for publication, and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Albert A has received honoraria speaking for Abbott Diagnostics on unrelated work. Haase M has received honoraria speaking for Abbott Diagnostics, Alere, Biosite Inc., and Siemens Healthineers. Albert C has received honoraria speaking for Siemens Healthineers. All other authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Albert C was supported by grants from the Dr. Werner Jackstädt Foundation, Wuppertal, Germany. Haase M was supported by grants from the German Heart Foundation (Deutsche Stiftung für Herzforschung, Frankfurt am Main, Germany) and from the Else Kröner-Fresenius-Stiftung, Bad Homburg, Germany.

  1. Hoste EAJ, Bagshaw SM, Bellomo R, Cely CM, Colman R, Cruz DN, et al. Epidemiology of acute kidney injury in critically ill patients: the multinational AKI-EPI study. Intensive Care Med 2015;41:1411-23.
    Pubmed CrossRef
  2. Nadim MK, Forni LG, Bihorac A, Hobson C, Koyner JL, Shaw A, et al. Cardiac and vascular surgery-associated acute kidney injury: the 20th International Consensus Conference of the ADQI (Acute Disease Quality Initiative) Group. J Am Heart Assoc 2018;7:e008834.
    Pubmed KoreaMed CrossRef
  3. Bellomo R, Ronco C, Kellum JA, Mehta RL, Palevsky P; Acute Dialysis Quality Initiative Workgroup. Acute renal failure - definition, outcome measures, animal models, fluid therapy and information technology needs: the Second International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) Group. Crit Care 2004;8:R204-12.
  4. Kidney Disease Improving Outcome, Acute Kidney Injury Working Group. KDIGO clinical practice guideline for acute kidney injury. Kidney Int Suppl 2012;2:1-141.
  5. Albert C, Albert A, Kube J, Bellomo R, Wettersten N, Kuppe H, et al. Urinary biomarkers may provide prognostic information for subclinical acute kidney injury after cardiac surgery. J Thorac Cardiovasc Surg 2018;155:2441-52.e13.
    Pubmed CrossRef
  6. Di Somma S, Magrini L, De Berardinis B, Marino R, Ferri E, Moscatelli P, et al. Additive value of blood neutrophil gelatinase-associated lipocalin to clinical judgement in acute kidney injury diagnosis and mortality prediction in patients hospitalized from the emergency department. Crit Care 2013;17:R29.
    Pubmed KoreaMed CrossRef
  7. Nickolas TL, Schmidt-Ott KM, Canetta P, Forster C, Singer E, Sise M, et al. Diagnostic and prognostic stratification in the emergency department using urinary biomarkers of nephron damage: a multicenter prospective cohort study. J Am Coll Cardiol 2012;59:246-55.
    Pubmed KoreaMed CrossRef
  8. Albert C, Haase M, Albert A, Kropf S, Bellomo R, Westphal S, et al. Urinary biomarkers may complement the Cleveland Score for prediction of adverse kidney events after cardiac surgery: a pilot study. Ann Lab Med 2020;40:131-41.
    Pubmed KoreaMed CrossRef
  9. Albert C, Zapf A, Haase M, Röver C, Pickering JW, Albert A, et al. Neutrophil gelatinase-associated lipocalin measured on clinical laboratory platforms for the prediction of acute kidney injury and the associated need for dialysis therapy: a systematic review and meta-analysis. Am J Kidney Dis 2020;76:826-41.e1.
    Pubmed KoreaMed CrossRef
  10. Ostermann M, Zarbock A, Goldstein S, Kashani K, Macedo E, Murugan R, et al. Recommendations on acute kidney injury biomarkers from the acute disease quality initiative consensus conference: a consensus statement. JAMA Netw Open 2020;3:e2019209.
    Pubmed CrossRef
  11. Habibzadeh F, Habibzadeh P, Yadollahie M. On determining the most appropriate test cut-off value: the case of tests with continuous results. Biochem Med 2016;26:297-307.
    Pubmed KoreaMed CrossRef
  12. Cho E, Kim SC, Kim MG, Jo SK, Cho WY, Kim HK. The incidence and risk factors of acute kidney injury after hepatobiliary surgery: a prospective observational study. BMC Nephrol 2014;15:169.
    Pubmed KoreaMed CrossRef
  13. Yang CH, Chang CH, Chen TH, Fan PC, Chang SW, Chen CC, et al. Combination of urinary biomarkers improves early detection of acute kidney injury in patients with heart failure. Circ J 2016;80:1017-23.
    Pubmed CrossRef
  14. Rozenfeld KL, Zahler D, Shtark M, Goldiner I, Keren G, Banai S, et al. Elevated neutrophil gelatinase-associated lipocalin for the assessment of structural versus functional renal damage among ST-segment elevation myocardial infarction patients. Blood Purif 2020;49:560-6.
    Pubmed CrossRef
  15. Pickering JW and Endre ZH. Linking injury to outcome in acute kidney injury: a matter of sensitivity. PLoS One 2013;8:e62691.
    Pubmed KoreaMed CrossRef
  16. Albert C, Haase M, Albert A, Zapf A, Braun-Dullaeus RC, Haase-Fielitz A. Biomarker-guided risk assessment for acute kidney injury: time for clinical implementation? Ann Lab Med 2021;41:1-15.
    Pubmed KoreaMed CrossRef
  17. Haase M, Haase-Fielitz A, Plass M, Kuppe H, Hetzer R, Hannon C, et al. Prophylactic perioperative sodium bicarbonate to prevent acute kidney injury following open heart surgery: a multicenter double-blinded randomized controlled trial. PLoS Med 2013;10:e1001426.
    Pubmed KoreaMed CrossRef
  18. Albert C, Albert A, Bellomo R, Kropf S, Devarajan P, Westphal S, et al. Urinary neutrophil gelatinase-associated lipocalin-guided risk assessment for major adverse kidney events after open-heart surgery. Biomark Med 2018;12:975-85.
    Pubmed KoreaMed CrossRef
  19. Grenier FC, Ali S, Syed H, Workman R, Martens F, Liao M, et al. Evaluation of the ARCHITECT urine NGAL assay: assay performance, specimen handling requirements and biological variability. Clin Biochem 2010;43:615-20.
    Pubmed CrossRef
  20. Hanley JA and McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29-36.
    Pubmed CrossRef
  21. Waikar SS, Betensky RA, Emerson SC, Bonventre JV. Imperfect gold standards for kidney injury biomarker evaluation. J Am Soc Nephrol 2012;23:13-21.
    Pubmed KoreaMed CrossRef
  22. Haase M, Devarajan P, Haase-Fielitz A, Bellomo R, Cruz DN, Wagener G, et al. The outcome of neutrophil gelatinase-associated lipocalin-positive subclinical acute kidney injury: a multicenter pooled analysis of prospective studies. J Am Coll Cardiol 2011;57:1752-61.
    Pubmed KoreaMed CrossRef
  23. Youden WJ. Index for rating diagnostic tests. Cancer 1950;3:32-5.
    Pubmed CrossRef
  24. Perkins NJ and Schisterman EF. The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. Am J Epidemiol 2006;163:670-5.
    Pubmed KoreaMed CrossRef
  25. Haase M, Bellomo R, Devarajan P, Schlattmann P, Haase-Fielitz A; NGAL Meta-analysis Investigator Group. Accuracy of neutrophil gelatinase-associated lipocalin (NGAL) in diagnosis and prognosis in acute kidney injury: a systematic review and meta-analysis. Am J Kidney Dis 2009;54:1012-24.
    Pubmed CrossRef
  26. The R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing. http://www.R-project.org (Updated on Mar 2023).
  27. Singer E, Elger A, Elitok S, Kettritz R, Nickolas TL, Barasch J, et al. Urinary neutrophil gelatinase-associated lipocalin distinguishes pre-renal from intrinsic renal failure and predicts outcomes. Kidney Int 2011;80:405-14.
    Pubmed KoreaMed CrossRef
  28. O'Neal JB, Shaw AD, Billings FT. Acute kidney injury following cardiac surgery: current understanding and future directions. Crit Care 2016;20:187.
    Pubmed KoreaMed CrossRef
  29. Thiele RH, Isbell JM, Rosner MH. AKI associated with cardiac surgery. Clin J Am Soc Nephrol 2015;10:500-14.
    Pubmed KoreaMed CrossRef
  30. Brenner H and Gefeller O. Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence. Stat Med 1997;16:981-91.
    CrossRef
  31. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007;115:928-35.
    Pubmed CrossRef
  32. Rodrigues CE and Endre ZH. Definitions, phenotypes, and subphenotypes in acute kidney injury-moving towards precision medicine. Nephrology 2023;28:83-96.
    Pubmed KoreaMed CrossRef
  33. Huen SC and Parikh CR. Molecular phenotyping of clinical AKI with novel urinary biomarkers. Am J Physiol Renal Physiol 2015;309:F406-13.
    Pubmed KoreaMed CrossRef
  34. Kellum JA and Devarajan P. What can we expect from biomarkers for acute kidney injury? Biomark Med 2014;8:1239-45.
    Pubmed KoreaMed CrossRef
  35. Zapf A, Albert C, Frömke C, Haase M, Hoyer A, Jones HE, et al. Meta‐analysis of diagnostic accuracy studies with multiple thresholds: comparison of different approaches. Biom J 2021;63:699-711.
    Pubmed CrossRef
  36. Jäntti T, Tarvasmäki T, Harjola VP, Pulkki K, Turkia H, Sabell T, et al. Predictive value of plasma proenkephalin and neutrophil gelatinase-associated lipocalin in acute kidney injury and mortality in cardiogenic shock. Ann Intensive Care 2021;11:25.
    Pubmed KoreaMed CrossRef
  37. Kim H, Hur M, Cruz DN, Moon HW, Yun YM. Plasma neutrophil gelatinase-associated lipocalin as a biomarker for acute kidney injury in critically ill patients with suspected sepsis. Clin Biochem 2013;46:1414-8.
    Pubmed CrossRef
  38. L'Acqua C, Sisillo E, Salvi L, Introcaso G, Biondi ML. Nephrocheck after cardiac surgery: does it play a role in daily practice? A sequel of "Nephrocheck results should be corrected for dilution.". Int J Artif Organs 2019;42:665-7.
    Pubmed CrossRef
  39. Albert C, Haase M, Albert A, Ernst M, Kropf S, Bellomo R, et al. Predictive value of plasma NGAL:Hepcidin-25 for major adverse kidney events after cardiac surgery with cardiopulmonary bypass: a pilot study. Ann Lab Med 2021;41:357-65.
    Pubmed KoreaMed CrossRef
  40. Greenland S. Avoiding power loss associated with categorization and ordinal scores in dose-response and trend analysis. Epidemiology 1995;6:450-4.
    Pubmed CrossRef