Article

Letter to the Editor

Ann Lab Med 2023; 43(1): 104-107

Published online January 1, 2023 https://doi.org/10.3343/alm.2023.43.1.104

Copyright © Korean Society for Laboratory Medicine.

Proposed Model for Evaluating Real-world Laboratory Results for Big Data Research

Sollip Kim, M.D., Ph.D.1 , Eun-Jung Cho, M.D., Ph.D.2 , Tae-Dong Jeong, M.D., Ph.D.3 , Hyung-Doo Park, M.D., Ph.D.4 , Yeo-Min Yun, M.D., Ph.D.5 , Kyunghoon Lee, M.D., Ph.D.6 , Yong-Wha Lee, M.D., Ph.D.7 , Sail Chun, M.D., Ph.D.1* , and Won-Ki Min, M.D., Ph.D.1,*

1Department of Laboratory Medicine, University of Ulsan College of Medicine and Asan Medical Center, Seoul, Korea; 2Department of Laboratory Medicine, Hallym University Dongtan Sacred Heart Hospital, Hwaseong, Korea; 3Department of Laboratory Medicine, Ewha Womans University College of Medicine, Seoul, Korea; 4Department of Laboratory Medicine and Genetics, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea; 5Department of Laboratory Medicine, School of Medicine, Konkuk University, Seoul, Korea; 6Department of Laboratory Medicine, Seoul National University Bundang Hospital and College of Medicine, Seongnam, Korea; 7Department of Laboratory Medicine and Genetics, Soonchunhyang University Bucheon Hospital, Soonchunhyang University College of Medicine, Bucheon, Korea

Correspondence to: Sail Chun, M.D., Ph.D.
Department of Laboratory Medicine, University of Ulsan College of Medicine and Asan Medical Center, 88 Olympic-ro 43-gil, Songpa-gu, Seoul 05505, Korea
Tel: +82-2-3010-4513, Fax: +82-2-478-0884, E-mail: sailchun@amc.seoul.kr

Won-Ki Min, M.D., Ph.D.
Department of Laboratory Medicine, University of Ulsan College of Medicine and Asan Medical Center, 88 Olympic-ro 43-gil, Songpa-gu, Seoul 05505, Korea
Tel: +82-2-3010-4503, Fax: +82-2-478-0884, E-mail: WKmin@amc.seoul.kr

*These authors equally contributed to this study.

Received: April 25, 2022; Revised: May 23, 2022; Accepted: July 27, 2022

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Dear Editor,

Real-world laboratory results are generated for clinical purposes rather than for research purposes. It is not possible to conduct proper big data research by combining non-validated real-world laboratory results. Although refining standardized terms and appropriate formats is essential to build big data with real-world laboratory results, data reliability is a more critical factor.

Many laboratory tests lack standardization or harmonization, and results vary among laboratories. Even if test items are standardized or harmonized, big data results may be biased if unreliable test results are included. For example, with the well-standardized HbA1c test, 5% of laboratories participating in a European trial reported a result exceeding the quality target from the actual value [1]. The reliability of laboratory results can also change over time. Lot changes of calibrators and reagents are frequent challenges that limit a laboratory’s ability to produce consistent results over time [2]. When the HbA1c reagent lot was changed, the HbA1c result increased by 0.5% (5.5 mmol/mol) on average [3]. In the case of non-standardized or non-harmonized tests, results may vary according to different test principles, or among different instruments or reagents even within the same test principles.

It is therefore a challenge to select research-level reliable real-world laboratory results obtained for a clinical purpose for secondary use in big data analysis. We propose reanalyzing the external quality assessment (EQA) model for evaluating real-world laboratory results for big data research (Fig. 1).

Figure 1. EQA results reanalysis model for determining test results that are sufficiently reliable for big data research. *“Harmonized” refers to whether or not the laboratory test item is harmonized based on the “Harmonization Status of Measurand” on the ICHCLR website (https://www.harmonization.net/); A convenient and effective method of evaluation is to use cumulative data of external quality assurance for each laboratory. For the harmonized tests, if the evaluation results are within the acceptable criteria, the laboratory results could be considered to be sufficiently reliable for big data research and data aggregation. For non-harmonized tests, if the evaluation results are within acceptable criteria, the laboratory results could be sufficiently reliable for big data research and data aggregation, but only within the peer group.
Abbreviations: EQA, external quality assessment; ICHCLR, International Consortium for Harmonization of Clinical Laboratory Results.

The first step is to check if the laboratory tests are harmonized because the results of different institutions should not be recklessly aggregated otherwise. The “International Consortium for Harmonization of Clinical Laboratory Results” provides a resource center for global activities to harmonize and standardize clinical laboratory measurement procedures [4]. The second step is to evaluate whether the laboratory results are sufficiently reliable using the laboratory’s cumulative EQA results. EQA providers accumulate data regarding each test’s standardization or harmonization level in participants [5, 6]. Reanalyzing the EQA results of participants could be the most convenient and useful method for this purpose. For harmonized tests, if the evaluation results are within the acceptance criteria, the laboratory results could be considered sufficiently reliable for data aggregation. For non-harmonized tests, data aggregation would only be suitable within the peer group, comprising laboratories that use the same methods or analyzers from the same manufacturer [6]. We suggest basing the evaluation on the following principles and verify the detailed protocol; the results should be transparently disclosed by each EQA provider (Fig. 2).

Figure 2. Five principles and two prerequisites for the proposed EQA results reanalysis model for evaluating test results suitable for big data research.
Abbreviation: EQA, external quality assessment.

Principle 1: Cumulative EQA data (e.g., data collected over several years) should be used for evaluation to reflect the laboratory’s reliability over time, because EQA reflects only the performance of a laboratory at a certain time point, whereas big data analysis is based on longitudinally collected data. It is preferable to include at least 24 samples to enable minimal imprecision evaluation [7].

Principle 2: Set the acceptance criteria as the total error. The total error is derived from bias and imprecision [7]. Bias can be calculated based on the target value, participants’ mean value, or the peer group mean value of the multi-year EQA according to standardization or harmonization. Imprecision could be calculated as the new protocol for the HbA1c imprecision certificate by the International Federation of Clinical Chemistry and Laboratory Medicine [8]. The acceptance criteria could be outcome-based, biological variation-based, or state-of-the art–based according to the characteristics of the test item.

Principle 3: Evaluate each test item. The performance of each test differs for each test item even in the same laboratory.

Principle 4: Evaluate periodically (e.g., annually); even in the same laboratory, instruments/reagents can change over time, or the level of quality control may vary.

Principle 5: Take a conservative approach for evaluation (i.e., evaluate strictly). If big data research is biased by including unreliable results, significant side effects (e.g., incorrect medical practices based on incorrect research results) can occur in the long run.

Two basic laboratory quality conditions should be observed as prerequisites: only an appropriate laboratory level (e.g., accredited laboratory) and only the results generated by the instruments/reagents authorized by the regulatory body should be included.

We emphasize the reliability of laboratory test results in data quality management for big data research and propose reanalyzing the EQA model for evaluating real-world laboratory results for big data research. Simply declaring an assessment of data quality in a research paper will not solve the quality issue of big data research using real-world laboratory results. We recommend certification or management of data quality at the national/organizational level. Our model based on cumulative EQA data offers a good option for evaluating medical institutions with real-world laboratory results that can be included in big data research. Since this article does not present a specific methodology, a more in-depth statistical approach is needed in the future to fully address this issue.

Cho EJ, Lee KH, and Jeong TD conceptualized the reanalysis model. Park HD, Yun YM, and Lee YW reviewed the manuscript. Chun S and Min WK designed and supervised the study. Kim S wrote the manuscript draft. All authors read and approved the final manuscript.

This study was supported by a research fund from the Korean Association of External Quality Assessment Service (Fund No. KEQAS-2020-01).

  1. EurA1c Trial Group. EurA1c: the European HbA1c trial to investigate the performance of HbA1c assays in 2166 laboratories across 17 countries and 24 manufacturers by use of the IFCC model for quality targets. Clin Chem 2018;64:1183-92.
    Pubmed CrossRef
  2. Thompson S and Chesher D. Lot-to-lot variation. Clin Biochem Rev 2018;39:51-60.
  3. Thaler MA, Iakoubov R, Bietenbeck A, Luppa PB. Clinically relevant lot-to-lot reagent difference in a commercial immunoturbidimetric assay for glycated hemoglobin A1c. Clin Biochem 2015;48:1167-70.
    Pubmed CrossRef
  4. ICHCLR. The International Consortium for Harmonization of Clinical Laboratory Results. https://www.harmonization.net (Updated on Apr 2022).
  5. Jeong TD, Cho EJ, Lee K, Lee W, Yun YM, Chun S, et al. Recent trends in creatinine assays in Korea: long-term accuracy-based proficiency testing survey data by the Korean Association of External quality assessment service (2011-2019). Ann Lab Med 2021;41:372-9.
    Pubmed KoreaMed CrossRef
  6. Kim S, Lee K, Park HD, Lee YW, Chun S, Min WK. Schemes and performance evaluation criteria of Korean Association of External Quality Assessment (KEQAS) for improving laboratory testing. Ann Lab Med 2021;41:230-9.
    Pubmed KoreaMed CrossRef
  7. Weykamp C, John G, Gillery P, English E, Ji L, Lenters-Westra E, et al. Investigation of 2 models to set and evaluate quality targets for hb a1c: biological variation and sigma-metrics. Clin Chem 2015;61:752-9.
    Pubmed KoreaMed CrossRef
  8. IFCC (International Federation of Clinical Chemistry and Laboratory Medicine). IFCC network for standardization of HbA1c. https://ifcchba1c.org/ (Updated on May 2022).