Practical Considerations for Clinical Laboratories in Top-down Approach for Assessing the Measurement Uncertainty of Clinical Chemistry Analytes
2022; 42(6): 630-637
Ann Lab Med 2021; 41(1): 51-59
Published online January 1, 2021 https://doi.org/10.3343/alm.2021.41.1.51
Copyright © Korean Society for Laboratory Medicine.
Tingting Li , M.D.1, Shunwang Cao , Ph.D.1, Yi Wang , M.D.1, Yujuan Xiong , Ph.D.1, Yuting He , M.D.2, Peifeng Ke , M.D.1, and Xianzhang Huang, Ph.D.1
1Department of Laboratory Medicine, The Second Affiliated Hospital of Guangzhou University of Chinese Medicine, Guangzhou, China; 2Department of Laboratory Medicine, The First Affiliated Hospital, Sun Yat-Sen University, Guangzhou, China
Correspondence to: Xianzhang Huang, Ph.D.
Department of Laboratory Medicine, The Second Affiliated Hospital of Guangzhou University of Chinese Medicine, 111 Dade Road, Guangzhou 510120, China
Tel: +86-020-81887233
Fax: +86-020-81867705
E-mail: huangxz020@gzucm.edu.cn
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Background: A small shift in high-sensitivity cardiac troponin T (hs-cTnT) assays can lead to different result interpretation and consequent patient management. We explored whether a small bias could be detected using conventional internal quality control (QC) procedures, evaluated the performance of moving average (MA)-based QC procedures, and proposed a new QC procedure based on the moving rate (MR) of positive patient results of hs-cTnT assays.
Methods:The ability of conventional QC to detect a 5 ng/L bias was examined using the13s/22s/R4s multi-rule procedure as deviation rules.We developed MA and MR procedures for the hs-cTnT assay using eight months of patient data. The performance of different MA or MR procedures was investigated by calculating the median number of patient samples affected until a bias introduced into the dataset was detected (MNPed). After comparing the MNPed across different procedures, we selected an optimal MA or MR procedure for validation. Validation graphs were plotted using the minimum, median, and maximum number of results affected until bias detection.
Results: Our conventional QC procedures could not detect a positive bias of 5 ng/L. When a positive bias was introduced, MNPed was much higher using MA than using MR, with cut-off values of 5 ng/L and 14 ng/L, respectively. MR validation charts for optimal procedures provided insight into the MR performance.
Conclusions: The MR procedure could detect different errors with few false alarms. In the hs-cTnT assay, the MR procedure with a smaller cut-off value outperformed MA and conventional QC procedures for small bias detection.
Keywords: Internal quality control, Positive bias, High-sensitivity troponin T, Moving average, Moving rate
Internal quality control (QC) plays a significant role in a clinical laboratory. Conventional QC procedures rely on the periodic measurement of commercial QC materials with a known concentration prepared in matrices; however, these artificial serum matrices may substantially differ from patient-derived serum samples [1]. Despite the numerous QC procedures for guiding laboratories to determine the most appropriate QC testing frequency [2], in China, many laboratories analyze only two concentrations of QC materials once a day [3].This frequency may be insufficient to rapidly detect analytical errors [4].
In response to these shortcomings,another QC procedure was proposed based on patient results, which can provide real-time error monitoring [5–12]. This patient-based real-time QC (PBRTQC) is performed by monitoring patient population parameters, such as the mean, median, or standard deviation (SD) of analyte values [11,13,14]. PBRTQC does not require the use of QC materials and testing, and instead relies only on calculations using existing patient results, thus avoiding commutability problems and minimizing cost. Fleming,
In 2015, the European Society of Cardiology guidelines produced rule-in and rule-out algorithms using high-sensitivity cardiac troponin T (hs-cTnT) assays in patients with suspected non-ST segment elevation myocardial infarction [17]. For such assays, even a small shift can lead to quite different interpretations of results and consequent patient management, thereby requiring further invasive investigations and corrective actions. Therefore, new QC procedures other than conventional QC or MA QC proceduresare needed to detect a small critical bias for highly sensitive assays such as the Elecsys hs-cTnT assay (Roche Diagnostics, Mannheim, Germany), which is used in most laboratories. Liu,
We explored whether conventional QC procedures, the MA procedure, or a new PBRTQC procedure based on the MR of positive patient results could be used to rapidly detect a small critical bias (e.g., 5 ng/L) for the hs-cTnT assay. This is the first study to comprehensively explore the ability of the MR procedure to detect biases of different sizes and to use bias detection curves and validation graphs for comparing MR procedures with different parameters and validate the performance of the optimal MR procedure.
We retrospectively reviewed 37,047 hs-cTnT assay results during eight months (January 1 to August 30, 2019) after excluding the results from 56 research samples, five proficiency samples, and 496 internal QC samples from the laboratory information system (LIS) of the Second Affiliated Hospital of Guangzhou University of Chinese Medicine, Guangzhou, China. We then organized the hs-cTnT assay results according to the reporting time to the LIS. hs-cTnT concentrations were measured from patient plasma samples in our hospital using Elecsys on the Roche Cobase 602 analyzer (Roche Diagnostics). There were three out-of-control points, which triggered the 13s rule (a run is rejected when a single control measurement exceeds the mean±3 SD control limit) over the eight months. One of the out-of-control points was due to the degradation of subpackaged QC material; to resolve this issue, we tested the new redissolved QC material. The other two out-of-control points were caused by bubble formation in controls during sample preparation; therefore, we retested the controls and retrained the new employee to avoid this issue. There was no out-of-control point caused by the hs-cTnT analyzer, method, or reagent in the internal QC chart using the 13s/22s/R4s multi-rule procedure [18] in our laboratory, and the clinicians did not question the hs-cTnT concentrations obtained. The hs-cTnT concentrations exhibited skewed distribution. All data were analyzed using Microsoft Excel 2007 (Microsoft, Redmond, Washington, USA). Since fewer samples were obtained on weekends and holidays, the daily run size of hs-cTnT assay was calculated only for working days. Overall, 29,989 patient results were obtained on 167 working days from January to August, resulting in a daily run size of 180 patient results.
The commercial QC materials were produced by Bio-Rad Laboratories (Hercules, CA, USA). In our laboratory, two concentrations of QC materials were analyzed once a day. The means of eight months of QC results for the two concentrations of QC materials were 254 ng/L and 995 ng/L, and the analytical coefficients of variation (CVa, calculated as SD/mean) were 3.9% and 4.0%, respectively. The ability of conventional QC procedure to detect a 5 ng/L bias at these two concentrations was then examined using the13s/22s/R4s multi-rule procedure as deviation rules.
To obtain the probability of a positive bias of 5 ng/L triggering one QC rule, the standard z-value was first calculated. First, we assumed that the QC concentrations follow a Gaussian distribution. Then, the standard z-value for the probability of obtaining a QC concentration larger than the N SD (N=1, 2, or 3) in the presence of a critical bias was calculated as follows [16]:
where Meanold and Meannew denote the mean concentration of a specific QC result before and after introducing critical bias, respectively.
Next, the probability of obtaining a QC concentration smaller than N SD can be obtained by referring to the z-table, which is denoted as
Finally, we would select an appropriate low QC concentration (denoted as Meanlow) that could detect a small critical bias of 5 ng/L. For example, if the desired power of detection for critical bias is set to 95% and the QC rule is 12s, then the low QC concentration could be calculated as:
Rearranging the above formula, we obtain:
where z=−1.65 for a 95% power of detection.
As an example, we illustrate the method of obtaining the probability of triggering 41S for a low concentration of 254 ng/L. First, we calculate the standard z-value as follows:
Based on the z-table, the p-value is 0.6879; therefore, the probability of triggering the 11S rule is 1–
To choose the appropriate low QC concentration for detecting the positive bias of 5 ng/L, we assumed that the CVa was the smaller of 3.9% and 4.0% (i.e., 3.9%). Hence, the low QC concentration was calculated as:
The MA QC procedure settings can be divided into three parts: (1) inclusion and exclusion criteria by applying truncation limits to exclude all values that are above or below a certain threshold; (2) MA calculation method, including the MA calculation algorithm and the block size,defined as the number of patient results to average in the algorithm; and (3) control limits.
When there were no truncation limits in the MA QC procedure, “extreme” patient results (e.g., critical values) that differed substantially from the mean analyte concentration could not be excluded, which would lead to a higher false rejection rate [11, 13, 15]. In our preliminary analysis, where no truncation was used, the false rejection rate reached up to 23%. Since hs-cTnT concentrations higher than 150 ng/L are considered critical values in our laboratory, these results were truncated to improve the performance of the MA procedure. The false rejection rate was calculated as the proportion of unaffected results that fell outside the control limit. Approximately 19% of the raw data were truncated, which were considered excessive, and block sizes of 20, 50, 100, 200, 500, and 1,000 were investigated. MA was then calculated as:
where Z(i) denotes the calculated mean for hs-cTnT result i, and X(i) denotes the hs-cTnT result for sample i. The MA procedure was operated in continuous mode, so that a new MA value was calculated for every new hs-cTnT assay result. The control limits were set using the mean and SD of the MA as follows:
The MR QC procedure settings can be also divided into three parts, like the MA procedure. However, because of the robustness of the MR QC procedure algorithm, no truncation limits are used in this QC procedure.
For the MR QC procedure, the hs-cTnT concentrations should be converted into a binary status (negative=hs-cTnT concentration below the cut-off value; positive=hs-cTnT concentration above the cut-off value). We chose medical decision points of hs-cTnT concentrations [17] (5, 14, and 52 ng/L) as cut-off values. After converting negative and positive results to 0 and 1, respectively, we calculated the MR of 1 (denoting positive results) in block sizes of 20, 50, 100, 200, 500, or 1,000 as follows:
where MR(i) denotes the calculated moving rate for hs-cTnT result i and T(i) denotes the converted binary value (1 or 0) of the hs-cTnT concentration for sample i. A new MR value was obtained for every new hs-cTnT assay result with the MR procedure operated continuously. The control limits were set using the mean and SD of the MR for positive results as follows:
MA or MR bias detection simulation and validation were performed as described by van Rossum and Kemperman [9,10]and van Rossum [19]. Bias detection simulation was conducted to investigate the performance of different MA or MR procedures by calculating the median number of patient samples affected until a bias introduced into the dataset was detected (MNPed). Bias detection was defined as an MA or MR value falling outside of control limits. Biases ranging from 1 ng/L to 150 ng/L were introduced at nine random positions and differed by at least 2,000 assay results. In addition, before bias introduction, all MA or MR procedures were run for 2,000 hs-cTnT raw results. The number of patient samples affected by the introduced bias was then calculated as the number of patient samples falling between the point of bias introduction and the first MA or MR value exceeding the control limits. Maximum, minimum, and median (i.e., MNPed) numbers of patient samples needed for bias detection were determined. After comparing the MNPed across different MA or MR procedures with different block sizes, we selected an optimal MA or MR procedure for validation.
Bias detection curves and validation graphs were plotted as described by van Rossum and Kemperman [9, 10], with the introduced error on the X-axis and the MNPed on the Y-axis. Validation graphs were generated by plotting MNPed as bars, with the introduced error on the X-axis and the number of results needed for bias detection on the Y-axis; the maximum and minimum number of patient samples affected until bias detection was plotted as error bars. The validation graph was used to reflect the performance of a specific QC procedure. For PBRTQC procedures, we expected the error to be detected between the daily scheduled internal QC measurements. Therefore, we compared the MNPed with the average daily run size.
The probabilities of triggering different QC procedures when a positive critical bias of 5 ng/L appeared are summarized in Table 1.
Parameters of the MA or MR procedures are presented in Table 2. We did not investigate QC procedures for cases with the CVa larger than 20% owing to the wide control limits. The false rejections of these PBRTQC procedures were all <1%, with some even lower than 0.1%.
To compare bias detection for a small systematic error corresponding to 5 ng/L hs-cTnT across MA or MR procedures, we used the MNPed values (Table 3). The MNPed of MA procedures with a corresponding block size were much larger than those of MR procedures when using a cut-off value of 5 ng/L or 14 ng/L. In addition, the MNPed of MR procedures with a cut-off value of 52 ng/L were all larger than 2,000.
As shown in Fig. 1, the MNPed of various QC procedures were plotted against increasing error. Based on these plots,we chose a block size of 100 for the MAand MR procedures with cut-off values of 5 and 14 ng/L, respectively, and a block size of 200 for MR procedures with a cut-off value of 52 ng/L as the optimal QC procedures. A randomly selected MR procedure to detect the introduced bias for hs-cTnT concentration measurement is presented in Fig. 2, which illustrates how the MR QC procedure detects a systematic error.
As shown in Fig. 3, in four optimal PBRTQC procedures, the detection of negative and positive bias was not equal. For the MA procedure, the MNPed increased again as bias grew larger than −20 ng/L. However, for the MR procedure, MNPed decreased with the increase in the negative bias and became constant when the positive bias was greater than the cut-off value. Hence, in the three optimal MR procedures, MR with a cut-off value of 52 ng/L could obtain the smallest MNPed for positive bias, while MR with a cut-off value of 5 ng/L could detect a very small bias (such as 5 ng/L) faster.
When comparing the MNPed with the average daily run size, only a positive bias of ≥20 ng/L was consistently detected by the selected MA procedure. The possibility of detecting a bias of 10 ng/L within a given day was 50% (Fig. 3A). For the MR procedure with a cut-off value of 5 ng/L (Fig. 3B), a positive bias of ≥5 ng/L and a negative bias of ≥−10 ng/L were consistently detected. There was an approximately 50% chance of detecting a bias of −8, −7, and −6 ng/L within a day. For the MR procedure with a cut-off value of 14 ng/L, a positive bias of ≥7 ng/L and a negative bias of ≥−50 ng/L were consistently detected. There was an approximate 50% chance of detecting a bias ranging from −16 to −10 ng/L within a day (Fig. 3C). For the MR procedure with a cut-off value of 52 ng/L, only a positive bias of ≥40 ng/L was consistently detected (Fig. 3D).
We demonstrated that the MR procedure with different cut-off values enables the rapid detection of different critical biases with a small chance of false alarms. MR procedures with a small cut-off value outperformed MA and conventional QC procedures for the rapid detection of a small bias.
Based on these results, there was only a small possibility (<10%) of detecting a positive bias of 5 ng/L by conventional QC procedures. To detect such small biases at 95% probability, the concentration of the QC material should be lower than 32.5 ng/L, which is often lower than that of available commercial QC materials. Consequently, laboratories may need to prepare for their QC samples in-house to meet the low concentration required. This requires pooling of patient samples, necessitating significant logistic planning and evaluation of stability and homogeneity, which may be difficult for some laboratories [20]. Like our findings for hs-cTnT, Liu,
Although PBRTQC procedure has not been widely used in clinical laboratories, slow but continuous improvements have been made in its understanding and methodology, especially the MA procedure, corresponding with efforts toward achieving continuous analytical quality assurance in the last 50 years [21]. Recently, Ng,
Although other PBRTQC procedures, such as the average of delta, moving median analysis, and the exponentially weighted MA [7, 14, 23, 24],were not investigated in this study, the developed MR procedure not only rapidly detected a small critical bias (5 ng/L) but also showed that the MNPeds for detecting larger bias were smaller than the average daily run size. These MR procedures employed existing patient results and were easy to implement. Since hs-cTnT concentrations were converted into a binary status, the MR procedure was sensitive to changes in concentration close to the cut-off value, while being robust against excessively large concentrations. Therefore, we considered it unnecessary to use optimized truncation limits, allowing the inclusion of all the results.
MR procedures with a small cut-off value showed superior detection of a small bias but required more results to detect a large bias. In contrast, MR procedures with a large cut-off value showed superior detection of a large bias but could not detect a very small critical bias. Preferably, QC procedures should be able to detect all critical biases. For this reason, MR procedures with several cut-off values may address the needs of a robust QC system, like Westgard’s multi-rules with different performances [18]. In this study, we chose only three medical decision points derived from clinical guidelines [17] as the cut-off values. In the future, we will design a method for selecting an appropriate cut-off value for the MR procedure to meet the requirement of patient safety assurance.
The main limitation of our study was that we implemented the MR procedure for hs-cTnT assay only and did not include other analytes that might have a different distribution. Another limitation of our strategy is that we could not exclude the possibility of undetected systematic errors in our dataset, which might decrease the sensitivity of the QC procedure. Ideally, the raw data in a dataset are obtained after maintenance and calibration of instruments to ensure that the analyzer is in a good state for providing reliable results. Since we used Excel software, we performed only nine bias detection simulations on the dataset. Therefore, the obtained results, including the maximum, minimum, and median concentrations, may not be accurate, especially for some results with larger error bars. Clearly, more accurate data would have been obtained with more simulations performed, which further motivates us to develop a software that can be used to design and optimize the MR procedure (e.g., determining the need for truncation limits).
Given the high complexity of testing environments in clinical laboratories, the use of a single QC procedure may not be enough to detect different types of errors. We suggest considering PBRTQC procedure when the performance of the conventional QC procedure is limited, which was also recommended by van Rossum and van den Broek [12]. MA-based QC procedure could be examined and optimized using the web-based application, MA Generator (https://www.huvaros.com/) [12, 25]. However, laboratories are limited to using Excel to determine MR QC parameters owing to a lack of specific software for this purpose. After integrating the parameters and calculation algorithm into LIS, we will obtain a QC chart like that shown in Fig. 2.
We showed that the MR procedure is superior to the MA procedure in detecting a small bias. Use of the MR procedure with a smaller cut-off value may detect a small bias more rapidly. Although MR procedure with a larger cut-off value may detect a large bias, it may not effectively detect a small bias. Hence, MR procedures can be designed with different cut-off values to detect different biases more rapidly.
Bias detection curves for hs-cTnT MA and MR procedures. Median number of patient samples affected until error detection (MNPed) needed for bias detection using (A) MA procedure, (B) MR procedure with a cut-off value of 5 ng/L, (C) MR procedure with a cut-off value of 14 ng/L, and (D) MR procedure with a cut-off value of 52 ng/L. For the MA curves, 3 and 150 ng/L were used as lower and upper truncation limits, respectively. Numbers in the keys within each panel represent the applied block size for the MA or MR calculation. The dashed line represents the average daily run size of 180 patient results.
Abbreviations: hs-cTnT, high-sensitivity cardiac troponin T; MA, moving average; MR, moving rate.
Graphical illustration of how MR QC procedure detects a systematic error. The horizontal green line in the middle of the graph represents the mean of the MR of positive patient results (negative results=hs-cTnT below 5 ng/L; positive results=hs-cTnT above 5 ng/L). The horizontal red lines represent the upper or lower control limits.
Abbreviations: hs-cTnT, high-sensitivity cardiac troponin T; MR, moving rate. QC, quality control.
Validation charts for selected optimal patient-based QC procedures for the hs-cTnT assay. (A) MA procedure,(B) MR procedure with a cut-off value of 5 ng/L, (C) MR procedure with a cut-off value of 14 ng/L, and (D) MR procedure with a cut-off value of 52 ng/L. The graphs show the median number of assay results needed for bias detection (MNPed) as bars, and the minimum/maximum number of results needed for bias detection as error bars. The introduced bias is plotted on the X-axis and the MNPed on the Y-axis. Graphs in panels A–C used block sizes of 100, and those in D used a block size of 200. The lower and upper truncation limits for MA procedure are 3 and 150 ng/L, respectively. The dashed line represents the average daily run size of 180 patient results.
Abbreviations: hs-cTnT, high-sensitivity cardiac troponin T; MA, moving average; MR, moving rate; QC, quality control.
Probabilities of triggering different QC rules when a positive bias of 5 ng/L appears at two QC concentrations
Concentration | 254 ng/L | 995 ng/L |
---|---|---|
Probability of triggering 11S rule | 31.21% | 19.22% |
Probability of triggering 12S rule | 7.08% | 3.07% |
Probability of triggering 22S rule | 0.50% | 0.10% |
Probability of triggering 13S rule | 0.71% | 0.21% |
Probability of triggering 41S rule | 0.95% | 0.14% |
Parameters of MA or MR procedures with different block sizes
PBRTQC procedures* | Block size | Mean | CV (%) | Lower control limit | Upper control limit | False rejection (%) |
---|---|---|---|---|---|---|
MA procedure | 100 | 21.39 ng/L | 16.75 | 10.59 ng/L | 32.14 ng/L | 0.63 |
200 | 21.37 ng/L | 13.45 | 12.68ng/L | 30.04 ng/L | 0.40 | |
500 | 21.36 ng/L | 11.22 | 14.17 ng/L | 28.47 ng/L | 0.86 | |
1,000 | 21.34 ng/L | 10.32 | 14.74 ng/L | 27.94 ng/L | 0.00 | |
MR procedure with a cut-off value of 5 ng/L | 100 | 75.82% | 10.52 | 51.89% | 99.76% | 0.40 |
200 | 75.83% | 8.82 | 55.78% | 95.89% | 0.48 | |
500 | 75.87% | 7.76 | 58.21% | 93.53% | 0.75 | |
1,000 | 75.90% | 7.31 | 59.24% | 92.55% | 0.00 | |
MR procedure with a cut-off value of 14 ng/L | 100 | 42.82% | 17.15 | 20.81% | 64.83% | 0.07 |
200 | 42.81% | 10.96 | 28.73% | 56.90% | 0.25 | |
500 | 42.81% | 8.13 | 32.38% | 53.26% | 0.04 | |
1,000 | 42.82% | 6.47 | 34.52% | 51.12% | 0.00 | |
MR procedure with a cut-off value of 52 ng/L | 200 | 18.27% | 20.00 | 7.20% | 29.35% | 0.01 |
500 | 18.28% | 15.43 | 9.82% | 26.74% | 0.01 | |
1,000 | 18.28% | 13.24 | 11.03% | 25.53% | 0.01 |
MNPed of different MA or MR procedures with different block sizes
Bias | Block size | PBRTQC procedures | |||
---|---|---|---|---|---|
MA procedure | MR procedure with a cut-off value of 5 ng/L | MR procedure with a cut-off value of 14 ng/L | MR procedure with a cut-off value of 52 ng/L | ||
+5 ng/L | 100 | 1,310 | 94 | 191 | ND |
200 | 1,806 | 132 | 212 | ND | |
500 | ND | 293 | 414 | ND | |
1,000 | ND | 534 | 580 | ND | |
−5 ng/L | 100 | ND | 221 | 366 | ND |
200 | ND | 201 | 446 | ND | |
500 | ND | 457 | 1,163 | ND | |
1,000 | ND | 848 | 972 | ND |