Article

Original Article

Ann Lab Med 2024; 44(6): 529-536

Published online June 26, 2024 https://doi.org/10.3343/alm.2024.0082

Copyright © Korean Society for Laboratory Medicine.

Quantitative Evaluation of the Real-World Harmonization Status of Laboratory Test Items Using External Quality Assessment Data

Sollip Kim , M.D., Ph.D.1, Tae-Dong Jeong , M.D., Ph.D.2, Kyunghoon Lee , M.D., Ph.D.3, Jae-Woo Chung , M.D., Ph.D.4, Eun-Jung Cho , M.D., Ph.D.5, Seunghoo Lee , M.D., Ph.D.1, Sail Chun , M.D., Ph.D.1, Junghan Song , M.D., Ph.D.3, and Won-Ki Min, M.D., Ph.D.1,6

1Department of Laboratory Medicine, University of Ulsan College of Medicine and Asan Medical Center, Seoul, Korea; 2Department of Laboratory Medicine, Ewha Womans University College of Medicine, Seoul, Korea; 3Department of Laboratory Medicine, Seoul National University College of Medicine and Seoul National University Bundang Hospital, Seongnam, Korea; 4Department of Laboratory Medicine, Dongguk University Ilsan Hospital, Goyang, Korea; 5Department of Laboratory Medicine, Hallym University Dongtan Sacred Heart Hospital, Hallym University College of Medicine, Hwaseong, Korea; 6Future Strategy Division, SD Biosensor, Seoul, Korea

Correspondence to: Won-Ki Min, M.D., Ph.D.
Department of Laboratory Medicine, University of Ulsan College of Medicine and Asan Medical Center, 88 Olympic-ro 43-gil, Songpa-gu, Seoul 05505, Korea
E-mail: wonkmin@gmail.com

Received: February 14, 2024; Revised: April 23, 2024; Accepted: June 20, 2024

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Background: In recent decades, the analytical quality of clinical laboratory results has substantially increased because of collaborative efforts. To effectively utilize laboratory results in applications, such as machine learning through big data, understanding the level of harmonization for each test would be beneficial. We aimed to develop a quantitative harmonization index that reflects the harmonization status of real-world laboratory tests.
Methods: We collected 2021–2022 external quality assessment (EQA) results for eight tests (HbA1c, creatinine, total cholesterol, HDL-cholesterol, triglyceride, alpha-fetoprotein [AFP], carcinoembryonic antigen [CEA], and prostate-specific antigen [PSA]). This EQA was conducted by the Korean Association of External Quality Assessment Service, using commutable materials. The total analytical error of each test was determined according to the bias% and CV% within peer groups. The values were divided by the total allowable error from biological variation (minimum, desirable, and optimal) to establish a real-world harmonization index (RWHI) at each level (minimum, desirable, and optimal). Good harmonization was arbitrarily defined as an RWHI value ≤ 1 for the three levels.
Results: Total cholesterol, triglyceride, and CEA had an optimal RWHI of ≤ 1, indicating an optimal harmonization level. Tests with a desirable harmonization level included HDL-cholesterol, AFP, and PSA. Creatinine had a minimum harmonization level, and HbA1c did not reach the minimum harmonization level.
Conclusions: We developed a quantitative RWHI using regional EQA data. This index may help reflect the actual harmonization level of laboratory tests in the field.

Keywords: Development, External quality assessment, Harmonization, Index, Laboratory results, Standardization

In laboratory medicine, “standardization” refers to the process of aligning methods and measurement procedures to an established set of standards, often set by international bodies [1]. Such alignment ensures that results from different laboratories are comparable and consistent, regardless of where or how the tests are conducted. In contrast, “harmonization” involves the achievement of equivalent reported values among methods and measurements, using, but not necessarily adhering to, a standard [1]. Harmonization focuses on minimizing differences in results generated by different laboratories and measurement procedures. The terms standardization and harmonization are frequently used interchangeably, as they share a common ultimate goal, i.e., to deliver laboratory results to stakeholders, such as clinicians and patients, that are comparable among laboratories and over time [2].

The importance of standardization and harmonization in healthcare is profound. These processes ensure the reliability and comparability of laboratory results, which are crucial for accurate diagnosis, treatment planning, and patient monitoring. Consistent results among laboratories enable healthcare providers to make informed decisions based on globally comparable data.

The International Consortium for Harmonization of Clinical Laboratory Results (ICHCLR) was established to drive the harmonization of results among different measurement procedures [3]. The ICHCLR prioritizes measurands by medical importance, coordinates the work of different organizations, and stimulates the development of technical and regulatory processes to achieve harmonization. The ICHCLR classifies the status of harmonization into the following categories: active, adequate/maintain, inactive, incomplete, and needed [4].

To objectively assess the harmonization status of each test, we hypothesized that quantitatively expressing the degree of harmonization would be more useful than qualitatively describing it as a “status,” as done by the ICHCLR. To our knowledge, no previous studies have quantitatively measured the standardization or harmonization of tests. Therefore, we quantitatively assessed the degree of standardization/harmonization of tests used in the real world using external quality assessment (EQA) data from the Korean Association of External Quality Assessment Service (KEQAS).

This study was exempt from institutional review board (IRB) approval as patient data were not collected (IRB No.: AMC 2023-1040). This study was conducted in accordance with the principles of the Helsinki Declaration and its amendments.

Collection of EQA data for simulation

In KEQAS proficiency tests (PTs), non-commutable QC materials can show a matrix effect in routine clinical chemistry parameters (e.g., electrolytes and proteins). Therefore, we selected tests that use commutable materials to exclude the matrix effect. Accuracy-based (AB)PTs, including HbA1c, creatinine, total cholesterol, HDL-cholesterol, and triglyceride, were the focus of this study. Additionally, tests for three tumor markers, namely alpha-fetoprotein (AFP), carcinoembryonic antigen (CEA), and prostate-specific antigen (PSA), were examined. For each test, a minimum of 12 EQA samples were used, with EQA data collected between January 2021 and December 2022.

Each HbA1c EQA material from a single donor was immediately aliquoted into vials and shipped on the same day at 4°C. The EQA materials for creatinine and lipids were produced with reference to the CLSI guideline C37-A, a reliable standard for creating commutable reference materials [5]. The EQA materials for tumor markers were prepared by spiking high-concentration patient samples into pooled serum from fresh frozen plasma and were evaluated and verified for commutability [6]. The data, which were divided into peer and sub-peer groups, adhered to the classification system established by KEQAS [7]. In KEQAS, for general chemistry, peer groups are based on the same methods and are further divided into reagent manufacturer-based sub-peer groups. For tumor markers, peer groups are based on laboratories using analyzers from the same manufacturer and are further subdivided into instrument- or reagent-based sub-peer groups.

Calculation of bias%, CV%, and total analytical error (TAE)%

According to the KEQAS evaluation guidelines, groups (peer or sub-peer) with less than 10 participating institutions or those with less than eight institutions post-outlier removal were omitted from the analysis; this is because KEQAS does not compute averages for these groups [7].

To calculate the average bias of peer groups or individual tests, the true value was employed for ABPTs. For non-ABPTs without a true value, calculations were performed using both the sub-peer group mean and the peer group mean. The peer group mean represents the average of the sub-peer group means within that peer group, whereas the overall mean is the average of the peer group means among all tests. In this study designed to observe harmonization among members of a peer group, we did not use the overall mean when there was a dominant sub-peer group, as doing so might have led to distorted results.

The CV% for each EQA sample was determined based on the results from the participating institutions, with the values averaged among the 12 samples. The peer group CV% was defined as the mean of the subgroup CV% within the peer group, and the overall CV% as the mean of the sub-peer group CV% for the entire tests.

The real-world (RW)-TAE% was calculated using the derived bias% and CV%, as follows: RW-TAE%=|bias%|+1.65*CV%. The factor 1.65 (one-sided estimate) implies that 95% of the results fall within the total allowable error (TEa) limit [8].

Calculation of the real-world harmonization index (RWHI) and comparison with the ICHCLR harmonization status

The RWHI was calculated using the following formula: RWHI= (RW-TAE%)/(biological variation [BV]-based TEa%).

The European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) BV-based minimum, desirable, and optimal (TEa%) levels for the measurements were retrieved from the EFLM BV database on December 21, 2023 [9]. Within these categories, “optimal” indicates no need for further assay improvement, “desirable” indicates satisfactory performance, and “minimal” indicates room for assay improvement [10]. The factors for optimum and minimum performance specifications are arbitrarily set to 1/2 and 3/2 of the desirable, respectively (in cases of imprecision, the factor is 0.25 for optimal, 0.5 for desirable, and 0.75 for minimum; in terms of bias, the values are 0.125 for optimal, 0.25 for desirable, and 0.375 for minimum) [9].

When the minimum, desirable, and optimal TEa% values were used as the denominator, the results were classified as minimum, desirable, and optimal RWHI, respectively. Minimum, desirable, and optimal RWHIs of ≤1 were arbitrarily considered to reflect the achievement of the corresponding harmonization levels. We compared the minimum, desirable, and optimal RWHI values with the harmonization status provided by the ICHCLR.

Basic characteristics of the data analyzed

The basic data for the eight tests, including the minimum and maximum number of institutions that participated in four sessions over 2 yrs for each test, are summarized in Table 1. True values for HbA1c (National Glycohemoglobin Standardization Program, NGSP), creatinine, total cholesterol, HDL-cholesterol, and triglyceride were used to calculate the TAE%. As true values were not available for AFP, CEA, and PSA, the peer group mean was used to calculate the TAE%.

Basic characteristics of the KEQAS EQA data
AnalyteUnit*Commutability of EQA materialsTarget value determinationN participating laboratories per peer group (minimum/maximum)Total participating
laboratories, N
HbA1c (NGSP)%CommutablePRMP500/5245,878
Creatininemg/dLCommutablePRMP1,445/1,53816,942
Total cholesterolmg/dLCommutablePRMP277/2953,255
HDL-cholesterolmg/dLCommutablePRMP261/2763,029
Triglyceridemg/dLCommutablePRMP97/1121,166
AFPμg/LCommutablePeer group mean632/6707,318
CEAμg/LCommutablePeer group mean515/5416,091
PSAμg/LCommutablePeer group mean500/5305,840

*Conversion factors of conventional units to SI units: HbA1c (IFCC [mmol/mol])=(10.93×NGSP [%])–23.50; creatinine, 1 mg/dL=88.4 μmol/L; total cholesterol, 1 mg/dL=0.0259 mmol/L; HDL-cholesterol, 1 mg/dL=0.0259 mmol/L; triglyceride, 1 mg/dL=0.0113 mmol/L.

EQA data were collected from January 2021 to December 2022.

Abbreviations: AFP, alpha-fetoprotein; CEA, carcinoembryonic antigen; EQA, external quality assessment; IFCC, International Federation of Clinical Chemistry; KEQAS, Korean Association of External Quality Assessment Service; PRMP, primary reference measurement procedure; PSA, prostate-specific antigen.



RWHI by tests based on the KEQAS EQA data

The minimum, desirable, and optimal RWHI, calculated using the RW-TAE% and EFLM BV-based minimum, desirable, and optimal TEa%, are presented in Table 2.

Real-world harmonization index for eight analytes from the KEQAS EQA data
AnalyteTEa from the EFLM database, %*ICHCLR dataKEQAS EQA data
Real-world TAE %RWHI by TEa specification
MinimumDesirableOptimalMedical impactHarmonization
status
BiasCVTAEMinimumDesirableOptimal
HbA1c (NGSP)3.32.21.1HighAdequate/maintain2.12.05.41.62.54.9
Creatinine11.17.43.7HighAdequate/maintain2.74.910.81.01.52.9
Total cholesterol12.98.64.3HighAdequate/maintain0.71.73.50.30.40.8
HDL-cholesterol16.410.95.5HighIncomplete6.22.710.60.61.01.9
Triglyceride40.527.013.5HighAdequate/maintain2.02.35.80.10.20.4
AFP26.517.68.8HighAdequate/maintain1.75.711.10.40.61.3
CEA30.820.510.3HighNeeded1.05.09.30.30.50.9
PSA24.416.28.1HighNeeded1.24.99.20.40.61.1

*The EFLM BV-based TEa% of measurement was retrieved from the EFLM BV database (https://biologicalvariation.eu/) on December 21, 2023.

The ICHCLR data were retrieved from the ICHCLR database (https://www.harmonization.net/) on December 18, 2023.

Abbreviations: AFP, alpha-fetoprotein; APS, analytical performance specification; CEA, carcinoembryonic antigen; EFLM, European Federation of Clinical Chemistry and Laboratory Medicine; EQA, external quality assessment; ICHCLR, International Consortium for Harmonization of Clinical Laboratory Results; KEQAS, Korean Association of External Quality Assessment Service; PSA, prostate-specific antigen; RWHI, real-world harmonization index; TAE, total analytical error; TEa, total allowable error.



The tests with an optimal harmonization level (optimal RWHI ≤1) were total cholesterol, triglyceride, and CEA. The tests with a desirable harmonization level (optimal RWHI>1 and desirable RWHI≤1) were HDL-cholesterol, AFP, and PSA. Creatinine showed a minimum harmonization level (desirable RWHI>1 and minimum RWHI≤1). HbA1c did not reach the minimal harmonization level based on our criteria.

RWHI by peer group in the accuracy-based KEQAS EQA program

The analysis results of RWHIs for the ABPT tests (HbA1c, creatinine, total cholesterol, HDL-cholesterol, and triglyceride) by peer group are presented in Table 3. For the HbA1c test, none of the peer groups reached the minimum harmonization level, as the minimum RWHI was not ≤1 in any of the groups. For creatinine, the enzymatic method peer group and rate-blanked compensated kinetic Jaffe method peer group had a minimum harmonization level; however, the kinetic Jaffe with compensation peer group and kinetic Jaffe without compensation peer group did not meet the minimum harmonization level criteria. As for HDL-cholesterol, the enzymatic with detergent, enzymatic with others, and enzymatic with polyethylene glycol peer groups had a desirable harmonization level, whereas the enzymatic with immuno-inhibition peer group achieved only a minimum harmonization level. Total cholesterol and triglyceride were not considered for peer group analysis because of the existence of only one peer group.

Real-world harmonization index by peer group in the accuracy-based KEQAS EQA program
AnalytePeer groupReal-world TAE, %RWHI by TEa specification
BiasCVTAEMinimumDesirableOptimal
HbA1c (NGSP)Abbott1.71.64.31.31.93.9
Arkray1.81.03.51.11.63.2
Bio-Rad2.01.34.11.21.93.7
Hitachi3.52.98.42.53.87.6
Roche2.11.85.11.62.34.7
Tosoh2.21.24.21.31.93.8
SD BIOSENSOR1.64.18.32.53.87.5
i-SENS1.92.25.61.72.65.1
CreatinineEnzymatic2.64.510.00.91.42.7
Kinetic Jaffe with compensation2.45.711.71.11.63.2
Kinetic Jaffe without compensation4.66.114.71.32.04.0
Kinetic Jaffe with rate-blanked and compensated1.33.26.70.60.91.8
Total cholesterolCholesterol esterase0.71.73.50.30.40.8
HDL-cholesterolEnzymatic with detergent4.02.37.70.50.71.4
Enzymatic with immuno-inhibition8.53.414.20.91.32.6
Enzymatic with others6.02.710.50.61.01.9
Enzymatic with polyethylene glycol6.32.29.90.60.91.8
TriglycerideEnzymatic with free glycerol elimination2.02.35.80.10.20.4

Abbreviations: EQA, external quality assessment; KEQAS, Korean Association of External Quality Assessment Service; RWHI, real-world harmonization index; TAE, total analytical error; TEa, total allowable error.



RWHI by peer group for tumor markers in the KEQAS EQA program

The analysis results of RWHIs for the three tumor marker tests (AFP, CEA, and PSA) by peer group are presented in Table 4. For AFP, the Abbott peer group had an optimal harmonization level, whereas the Beckman Coulter, Roche, and Siemens peer groups had a desirable harmonization level. CEA was associated with an optimal harmonization level in all peer groups. As for PSA, the Roche peer group achieved an optimal harmonization level, whereas the Abbott, Beckman Coulter, and Siemens peer groups had a desirable harmonization level.

Real-world harmonization index by peer group for tumor markers in the KEQAS EQA program
AnalytePeer groupReal-world TAE, %RWHI by TEa specification
BiasCVTAEMinimumDesirableOptimal
AFPAbbott0.42.74.80.20.30.5
Beckman Coulter1.35.911.10.40.61.3
Roche3.46.113.40.50.81.5
Siemens1.68.315.20.60.91.7
CEAAbbott1.14.58.50.30.40.8
Beckman Coulter0.05.99.70.30.50.9
Roche1.54.18.30.30.40.8
Siemens1.35.610.50.30.51.0
PSAAbbott1.65.09.80.40.61.2
Beckman Coulter0.06.09.90.40.61.2
Roche1.43.87.60.30.50.9
Siemens1.84.79.50.40.61.2

Abbreviations: AFP, alpha-fetoprotein; CEA, carcinoembryonic antigen; EQA, external quality assessment; KEQAS, Korean Association of External Quality Assessment Service; PSA, prostate-specific antigen; RWHI, real-world harmonization index; TAE, total analytical error; TEa, total allowable error.


Artificial intelligence (AI) can play a crucial role in modern healthcare, particularly in improving disease diagnosis, optimizing treatment, predicting prognosis and outcomes, developing drugs, and improving public health [11]. In the era of AI, big data are crucial and play a pivotal role in the advancement of AI technologies [12]. Big data facilitate effective model training and the discovery of complex patterns in diverse health datasets, thereby improving its accuracy in diagnosing conditions, identifying health trends, and creating personalized treatment plans.

Effective AI depends on access to substantial amounts of high-quality data; the source, size, and quality of the data can significantly influence the development of AI models [13]. Laboratory medicine is an essential element of the healthcare system, and laboratory data significantly contribute to objective clinical data, playing an integral role in numerous clinical decisions [14]. Therefore, ensuring standardized/harmonized laboratory data in big data, including structure, coding, and results, is essential for the effectiveness of AI in healthcare [15-17]. Harmonization enables clinical interoperability and supports machine learning using various platforms [18]. Non-harmonized tests should be interpreted separately to avoid misinterpretation [19]. Incorrect harmonization assumptions risk patient safety, and missing data hinder public health analysis [18].

To our knowledge, no previous studies have quantitatively measured the standardization/harmonization of tests to reflect their actual level of harmonization. Therefore, we used KEQAS EQA data to assess the degree of standardization/harmonization of laboratory tests in real-world settings. The differences with the existing ICHCLR harmonization levels [4] are provided in Table 2. Among the tests labeled “adequate/maintain” by the ICHCLR, including HbA1c, creatinine, AFP, total cholesterol, and triglyceride, the latter three achieved optimal harmonization. AFP reached a desirable level, creatinine achieved the minimum level, and the harmonization level of HbA1c was below the minimum. HDL-cholesterol, classified as “incomplete” by the ICHCLR, had a desirable harmonization level. CEA and PSA, labeled “needed” for harmonization in ICHCLR, had optimal and desirable levels, respectively.

The ICHCLR categorization is based on a comprehensive assessment, including evaluations of inter-laboratory CV%, inter-method CV%, and/or bias%, primarily using data from EQA programs. The assessment also considers the availability of reference materials, the presence of reference measurement procedures, and the traceability of commercially available calibrators. However, because of the various evaluation methods for each test used by the ICHCLR and the absence of a predefined schedule for timely data updates, assessing the current status of harmonization remains challenging. The harmonization status from the ICHCLR does not integrate and reflect the test methods used in the real world and does not allow quantitative comparison of the degree of standardization or harmonization of each test.

The discrepancy between our results and the ICHCLR harmonization status may be due to three factors. First, the ICHCLR evaluation criteria, such as the availability of reference materials and traceability assessment of calibrators, were not used in our study. Second, we used bias% and CV%, whereas the ICHCLR subjectively used overall EQA data, and we employed a quantitative average from peer group data. Finally, while the ICHCLR criteria were based on global data, we only used EQA results from Korea.

KEQAS differently classifies peer and sub-peer groups according to the tests. In ABPT tests, the peer group represents the test method, whereas the sub-peer group corresponds to the reagent manufacturer. In contrast, for tumor markers, the peer group represents the instrument manufacturer, and the sub-peer group pertains to the model of the instrument from the same manufacturer.

In the ABPT tests, which included HbA1c, creatinine, and HDL-cholesterol, variations were noted in the harmonization levels among peer groups. For HbA1c, Arkray, Bio-Rad, and Tosoh, which use high-performance liquid chromatography, had RWHI values between 1.1 and 1.3. In contrast, Abbott, Hitachi, and i-SENS, which use an enzymatic method, had RWHI values ranging from 1.3 to 2.5. Roche and SD BIOSENSOR, which use an immunoassay, had an RWHI ranging from 1.6 to 2.5. Although none of the methods met the minimum harmonization level, the high-performance liquid chromatography method can be considered better harmonized than the other test methods. Therefore, this method is a valuable reference during the selection of a test instrument. The creatinine test generally met the minimum harmonization level; however, a considerable variation, ranging from falling below the minimum level to achieving a desirable harmonization level, was found depending on the peer group. Therefore, laboratories using lower harmonization-level creatinine test methods should consider switching to higher harmonization-level test methods or opting for products with improved harmonization levels within the same test method. In terms of the HDL-cholesterol test, all peer groups, except the enzymatic with immune inhibition peer group at the minimum harmonization level, achieved a desirable harmonization level. Further studies are needed to investigate the underlying reasons for the differences among the test methods.

In KEQAS, the tumor marker peer group consists of four companies: Abbott, Beckman Coulter, Roche, and Siemens. AFP and PSA achieved a desirable harmonization level, with the Abbott peer group excelling in AFP and the Roche group excelling in PSA. A high harmonization level within a peer group indicates strong consistency in test results among devices, indicating robust harmonization, which may be associated with the types of instrument platforms and reagents. In the Roche peer group, a single platform (cobas e series) was employed. In the Abbott peer group, two platforms (Alinity I and Architect) were employed, but the reagents were the same. In contrast, Siemens had two platforms (Centaur and Atellica IM), as did Beckman Coulter (Access2 and UniCel DxI800).

The recent proposal of Test Result Harmonization Status for the United States Core Data for Interoperability (USCDI) by the College of American Pathologists, though not fully integrated into the USCDI, can serve as an important criterion for future big data applications [18]. Our quantitative approach to expressing harmonization levels may serve as the foundation. Currently, the RWHI calculations depend on TEa derived from biological variation. Therefore, to enhance interoperability with big data, future TEa determinations should be based on big data standards.

This study focused on eight test items using commutable materials without matrix effects among PT items with long-term data. Future research opportunities may arise if other EQA providers release results based on commutable materials for additional tests. The RWHI developed in our study is based on test methods and instrument results used in Korea. When EQA data from other countries are analyzed, different outcomes reflecting the harmonization level of each country are expected because of differences in the test methods and equipment used in those countries.

In conclusion, we introduced the concepts of real-world TAE% and RWHI and quantitatively assessed the degree of harmonization for eight tests using KEQAS EQA data. This index can be used to assess interoperability in future big data analyses as it reflects the actual harmonization level of laboratory tests in the field.

We thank the Korean Association of External Quality Assessment Service (KEQAS) for providing the data for the analysis.

Min WK contributed to study conceptualization; Jeong TD, Lee K, and Cho EJ contributed to the methodology, Kim S, Chung JW, and Lee S contributed to the investigation; Kim S and Min WK acquired the funding; Min WK administered the project, Chun S and Song J supervised the study; Kim S wrote the original draft; Kim S, Jeong TD, Lee K, Chung JW, Cho EJ, and Min WK reviewed and edited the manuscript. All authors accept responsibility for the entire content of this manuscript. All authors have read and approved the final manuscript.

This research was supported by the Quality Improvement Research Program funded by the Laboratory Medicine Foundation (LMF 2023-02).

  1. Miller WG, Greenberg N. Harmonization and standardization: where are we now?. J Appl Lab Med 2021;6:510-21.
    Pubmed CrossRef
  2. Plebani M, Lippi G. Standardization and harmonization in laboratory medicine: not only for clinical chemistry measurands. Clin Chem Lab Med 2023;61:185-7.
    Pubmed CrossRef
  3. Myers GL, Miller WGEJIFCC. The International Consortium for Harmonization of Clinical Laboratory Results (ICHCLR) - a pathway for harmonization. EJIFCC 2016;27:30-6.
  4. The International Consortium for Harmonization of Clinical Laboratory Results. https://www.harmonization.net (Updated on Apr 2024).
  5. CLSI. Preparation and validation of commutable frozen human serum pools as secondary reference materials for cholesterol measurement procedures. 1st ed. Wayne, PA: Clinical and Laboratory Standards Institute, 1999: 52.
  6. Cho EJ, Kim SH, Hong J, Lee H, Hyun J, Cho SE, et al. Commutability Assessment of Frozen Human Serum Pools for External Quality Assessment of Tumor Markers. Lab Med Qual Assur 2022;44:111-120.
    CrossRef
  7. Kim S, Lee K, Park HD, Lee YW, Chun S, Min WK. Schemes and performance evaluation criteria of Korean Association of External Quality Assessment (KEQAS) for improving laboratory testing. Ann Lab Med 2021;41:230-9.
    Pubmed KoreaMed CrossRef
  8. Oosterhuis WP, Bayat H, Armbruster D, Coskun A, Freeman KP, Kallner A, et al. The use of error and uncertainty methods in the medical laboratory. Clin Chem Lab Med 2018;56:209-19.
    Pubmed CrossRef
  9. Aarsand A, Fernandez-Calle P, Webster C, Coskun A, Gonzales-Lao E, Diaz-Garzon J, et al. The EFLM Biological Variation Database https://biolo.
  10. Jones GRD. Analytical performance specifications for EQA schemes - need for harmonization. Clin Chem Lab Med 2015;53:919-24.
    Pubmed CrossRef
  11. Noorbakhsh-Sabet N, Zand R, Zhang Y, Abedi V. Artificial intelligence transforms the future of health care. Am J Med 2019;132:795-801.
    Pubmed KoreaMed CrossRef
  12. Big data and AI: a comparative overview. https://innovatureinc.com/big-data-and-ai-a-comparative-overview (Updated on Apr 2024).
  13. Chen M, Decary M. Artificial intelligence in healthcare: an essential guide for health leaders. Healthc Manage Forum 2020;33:10-8.
    Pubmed CrossRef
  14. Hallworth MJ. The '70% claim': what is the evidence base?. Ann Clin Biochem 2011;48:487-8.
    Pubmed CrossRef
  15. Kim HS, Kim DJ, Yoon KH. Medical big data is not yet available: why we need realism rather than exaggeration. Endocrinol Metab (Seoul) 2019;34:349-54.
    Pubmed KoreaMed CrossRef
  16. Cho EJ, Jeong TD, Kim S, Park HD, Yun YM, Chun S, et al. A new strategy for evaluating the quality of laboratory results for big data research: using external quality assessment survey data (2010-2020). Ann Lab Med 2023;43:425-33.
    Pubmed KoreaMed CrossRef
  17. Kim S. Laboratory data quality evaluation in the big data era. Ann Lab Med 2023;43:399-400.
    Pubmed KoreaMed CrossRef
  18. United States Core Data for Interoperability. Laboratory Test Result Harmonization Status. https://www.healthit.gov/isa/uscdi-data/test-result-harmonization-status (Updated on Apr 2024).
  19. Kim S, Cho EJ, Jeong TD, Park HD, Yun YM, Lee K, et al. Proposed model for evaluating real-world laboratory results for big data research. Ann Lab Med 2023;43:104-7.
    Pubmed KoreaMed CrossRef