Muhammad Shafiq,Diego Robles Mazzotti, Cheryl Gibson

Abstract

BACKGROUND

Risk stratification tools exist for patients presenting with chest pain to the emergency room and have achieved the recommended negative predictive value(NPV) of 99%. However, due to low positive predictive value (PPV), current stratification tools result in unwarranted investigations such as serial laboratory tests and cardiac stress tests (CSTs).

AIM

To create a machine learning model (MLM) for risk stratification of chest pain with a better PPV.

METHODS

This retrospective cohort study used de-identified hospital data from January 2016 until November 2021. Inclusion criteria were patients aged > 21 years who presented to the ER, had at least two serum troponins measured, were subsequently admitted to the hospital, and had a CST within 4 d of presentation.Exclusion criteria were elevated troponin value (> 0.05 ng/mL) and missing values for body mass index. The primary outcome was abnormal CST.Demographics, coronary artery disease (CAD) history, hypertension, hyperlipidemia, diabetes mellitus, chronic kidney disease, obesity, and smoking were evaluated as potential risk factors for abnormal CST. Patients were also categorized into a high-risk group (CAD history or more than two risk factors)and a low-risk group (all other patients) for comparison. Bivariate analysis was performed using a χ2 test or Fisher’s exact test. Age was compared by t test.Binomial regression (BR), random forest, and XGBoost MLMs were used for prediction. Bootstrapping was used for the internal validation of prediction models. BR was also used for inference. Alpha criterion was set at 0.05 for all statistical tests. R software was used for statistical analysis.

RESULTS

The final cohort of the study included 2328 patients, of which 245 (10.52%) patients had abnormal CST. When adjusted for covariates in the BR model, male sex [risk ratio (RR) = 1.52, 95% confidence interval (CI): 1.2-1.94, P < 0.001)], CAD history (RR = 4.46, 95%CI: 3.08-6.72, P < 0.001), and hyperlipidemia (RR = 3.87, 95%CI: 2.12-8.12, P < 0.001) remained statistically significant. Incidence of abnormal CST was 12.2% in the high-risk group and 2.3% in the low-risk group (RR = 5.31,95%CI: 2.75-10.24, P < 0.001). The XGBoost model had the best PPV of 24.33%, with an NPV of 91.34% for abnormal CST.

CONCLUSION

The XGBoost MLM achieved a PPV of 24.33% for an abnormal CST, which is better than current stratification tools (13.00%-17.50%). This highlights the beneficial potential of MLMs in clinical decision-making.

Key Words: Machine learning; Chest pain; Risk stratification; Risk factors; Cardiac stress test; Cardiac catheterization

INTRODUCTION

The annual cost of cardiovascular disease and stroke has been estimated to be more than $200 billion in the United States[1]. Chest pain, in particular, led to over 40 million hospital visits from 2006-2016 and had an associated cost of $6.2 billion from 2014-2016[2]. For each hospital admission, the average cost has been reported to be $11700 per patient in the United States[3]. This cost represents a significant burden on the patients and the healthcare system and emphasizes the need for a better stratification tool to safely identify patients who present with chest pain for early discharge without unnecessary testing.

Patients who present with chest pain are considered to be low risk if their probability of a major adverse cardiac event is < 1%, according to clinical practice guidelines of the American College of Cardiology and American Heart Association[4]. The two most widely used stratification tools that calculate the probability of major adverse cardiac event include the history, electrocardiogram (ECG),age, risk factors, and initial troponin pathway accelerated diagnostic protocol (HEART pathway-ADP)and the emergency department assessment of chest pain score (EDACS)-ADP[5-8]. The HEART pathway-ADP and EDACS-ADP have demonstrated excellent sensitivity and a negative predictive value (NPV) of 99%, as recommended by the American College of Cardiology and the American Heart Association[5-8]. However, both the HEART pathway-ADP and EDACS-ADP have low positive predictive values (PPVs) of 13.0% and 17.5%, respectively[7,9]. This low PPV leads to unnecessary additional testing, such as serial laboratory tests and cardiac stress tests (CSTs). Furthermore, the main focus of these two stratification tools has been the 30-d outcome of acute myocardial infarction (AMI)and all-cause mortality.

Many medical devices are already using artificial intelligence and machine learning algorithms[10].However, these domains are underutilized in clinical decision-making despite their huge potential. The National COVID Cohort Collaborative facilitated the creation of machine learning models (MLMs) for accurate prediction of coronavirus disease 2019 severity with proven efficacy[11]. The same concept of machine learning can be applied to patients who present with chest pain in the emergency room (ER),and the use of patient data points (such as demographics and risk factors) can build predictive models with better specificity and PPV.

Stewartet al[12] reported in a recent systematic review of 23 studies that MLM outperformed traditional risk stratification scores for chest pain. Although the outcome varied among the included studies, data and the specifics of the MLMs, such as hyperparameters, were not shared, which complicates replication and validation[12]. Most recently, Doudesiset al[13] published the myocardialischemic-injury-index (MI3) algorithm with high PPV (70.4%) and NPV (99.8%) for myocardial infarction. They have not provided the specifics of their MLM either.

The HEART pathway-ADP, the EDACS-ADP, and other recently developed MLMs include both normal and abnormal troponin levels in their stratification[7,8,12,14,15]. However, risk stratification is more challenging when the troponin level is normal. In addition, the HEART pathway-ADP and the EDACS-ADP do not address the risk of non-obstructive coronary artery disease (CAD) or the risk of AMI over an extended period of time (over a year). Lastly, recently developed MLMs claim to have better stratification, but essential information about their models have not been shared, limiting the potential for reproducibility. Given these challenges to risk stratification among patients with chest pain and the need to improve patient outcomes and reduce unnecessary healthcare costs, we hypothesized that an MLM could be created to better predict abnormal CST among patients who present with chest pain and have normal troponin values. CST can identify wide spectrum of CAD, including nonobstructive CAD, and it provides risk assessment for 1 year rather than just 30 d. Due to the current enhanced computing capabilities, MLMs have the potential to achieve better PPV and lower falsepositive rates, which can reduce unnecessary testing. MLMs can gain the reproducibility needed to build trust through data sharing and transparency, which can further improve risk stratification and deliver the most cost-effective healthcare.

This study aimed to create an MLM that can use patient characteristics to provide risk stratification for further clinical intervention for patients who present with chest pain and have normal troponin values. The hypothesis of this study was that patient characteristics can be used to create MLMs for risk stratification for patients who present to the ER with chest pain and have normal troponin values with a PPV of 25% or more and an NPV of at least 99%.

MATERIALS AND METHODS

Study design

This study used a retrospective cohort design involving de-identified data available in the i2b2 common data model repository of the University of Kansas Medical Center. Database queries for patients observed in the health system between January of 2016 and November of 2021 were conducted using Healthcare Enterprise Repository for Ontological Narration, a search discovery tool that allows cohort building for observational research using de-identified data[16,17]. Institutional Review Board approval was not required because the data was de-identified. Identification and/or definitions of computable phenotypes used in this study are provided in the Supplementary material.

Selection criteria

Inclusion criteria:All patients aged 21 years or older who presented to the ER, had their first troponin test carried out within the first 6 h of arrival to the ER and at least one troponin test completed after 6 h,were subsequently admitted to the hospital, and had a nuclear CST carried out within 4 d of presentation to the ER were included in the study.

Exclusion criteria:Patients with elevated troponin levels (> 0.05 ng/mL) and those with missing body mass index values were excluded. Patients with elevated troponin levels are considered high risk, and hospital admission with further clinical intervention is more appropriate for this class of patients.Without body mass index data, obesity could not be defined as one of the risk factors/covariates in this study.

Risk stratification

CAD history, diabetes mellitus, hypertension, hyperlipidemia, chronic kidney disease, obesity, and smoking history were the risk factors included in this study. For comparison, patients included in the final cohort were also categorized into a high-risk group (CAD history or > 2 risk factors) and a low-risk group (no prior CAD history and ≤ 2 risk factors).

Encounters

Only the first encounter was included in the study for patients with more than one ER encounter that met the above inclusion and exclusion criteria.

Outcome

The outcome of the study was the incidence of abnormal CST. Abnormal CST was defined as CST followed by cardiac catheterization and/or coronary artery bypass graft within 30 d. Cardiac catheterization and coronary artery bypass graft were used as a surrogate to identify abnormal CST in this study.

Statistical analysis

Bivariate analysis:Except for age, the association between the incidence of abnormal CST and sex, race,and risk factors was assessed byχ2test or Fisher’s exact test. High-risk and low-risk groups were also compared using aχ2test, and the risk ratio was calculated. Age was compared byttest. Alpha criterion was set at 0.05.

Binomial regression:Binomial regression (BR) was used to adjust for confounding factors and infer the degree of association between the risk factors and the outcome. All BR assumptions were assessed,including assumptions of no multi-collinearity and no outliers. To ensure model adequacy, Hosmer-Lemeshow goodness-of-fit test was also performed. Alpha criterion both for BR as well as Hosmer-Lemeshow goodness-of-fit test was set at 0.05.

MLMs:BR was also used for prediction, and it used the same predictor variables as for inference.Besides BR, the random forest and XGBoost MLMs were also used. In the random forest MLM,proximity and importance were set as “True”. In order to minimize overfitting, the number of trees was set at 25 in training and testing. In the XGBoost MLM, hyperparameters were set as: Booster = “gbtree”;objective = “binary:logistic”; eval_metric = “auc”; eta = 0.1; max.depth = 10; gamma = 0; min_child_weight = 1; and colsample_bytree = 1. In order to minimize overfitting, 25 rounds were used in the XGBoost MLM.

Bootstrapping with replacement was used for internal validation of all the above models. Data were randomly split into training (75%) and testing (25%) during each iteration of bootstrapping. The model with training data was first fitted, and then testing data were applied to assess internal validation during each iteration of bootstrapping. Finally, the results were averaged. In order to produce more precise estimates, 500 iterations were used in bootstrapping for all models, and 95% confidence intervals(CIs) were created. Prediction cutoff values were calibrated manually, and the value with the best metrics (sensitivity, specificity, PPV, and NPV) was then selected for each model. All statistical analyses were performed using R software (version 4.1.2).

RESULTS

The final cohort sample included 2328 unique patients, of which 245 (10.52%) patients had abnormal CST requiring cardiac catheterization and/or coronary artery bypass graft (Figure 1). There were 196 duplicate encounters, which were removed, and only the patient’s first encounter was included in the study. The basic demographic characteristics of the patients are presented in Table 1. Male sex and Caucasian race were significantly associated with the incidence of abnormal CST.

Table 1 Demographics

Figure 1 Flowchart of the cohort selection. BMI: Body mass index; CABG: Coronary artery bypass graft; ER: Emergency room.

In the bivariate analysis, obesity was not significantly associated with abnormal CST. Smoking history was significantly associated with abnormal CST, but the association was weak. As shown in Table 2, all other risk factors were significantly associated with abnormal CST. High-risk patients were found to have a risk ratio of 5.31 (95%CI: 2.75-10.24) for abnormal CST when compared to low-risk patients (Table 3).

Table 2 Bivariate analysis of the risk factors

Table 3 Comparison between the high-risk group and the low-risk group

Age, race, and obesity were removed from the final BR model because they were not statistically significant, did not have any confounding relationship with other covariates, and their contribution to the prediction was not significant. These three covariates together resulted in an increased area under the receiver operating characteristic curve (AUC) by only 1.35%. The final model met all assumptions of BR, including the assumption of no multi-collinearity and no outliers, and appeared to fit the data adequately (Hosmer-Lemeshow goodness-of-fit test,P= 0.9). Only male sex, CAD history, and hyperlipidemia were statistically significant when adjusted for covariates in the final BR model.Covariates that were included in the final BR model and their estimated risk ratios are shown in Table 4.

Table 4 Adjustment of risk factors association with the abnormal cardiac stress test via binomial regression

Prediction models

BR:The same covariates used for inference in the BR model were also used for prediction. These covariates included sex, CAD history, hypertension, hyperlipidemia, diabetes mellitus, chronic kidney disease, and smoking history. First, all data were used together, which yielded an AUC of 74.51%(Figure 2A). The internal validation of the BR model yielded similar results (Figure 2B). A prediction cutoff value of 0.2 provided a sensitivity of 45.06%, specificity of 80.46%, a PPV of 21.34%, and an NPV of 92.55%.

Figure 2 Receiver operating characteristic curve. A: The binomial regression (BR) model; B: The interval validation of the BR model; C: The random forest models with 25 trees and 50 trees; D: The internal validation of the random forest model; E: The XGBoost model; F: The internal validation of the XGBoost model.AUC: Area under the receiver operating characteristic curve.

Random forest:In the random forest model, all covariates were included. This model showed a much better fit for all data combined (Figure 2C). During internal validation of the random forest model, the AUC dropped significantly (Figure 2D). A prediction cutoff value of 0.18 provided a sensitivity of 13.92%, specificity of 93.66%, a PPV of 20.55%, and an NPV of 90.24 % for the random forest model.

XGBoost:All covariates were used in the XGBoost model. It provided a better AUC for all data combined compared to BR (Figure 2E). Like the random forest model, the AUC dropped significantly during internal validation of the XGBoost model (Figure 2F). A prediction cutoff value of 0.27 was used for the XGBoost model, and it yielded a sensitivity of 30.54%, specificity of 88.51%, a PPV of 24.33%, and an NPV of 91.34%. A comparison of the three models is presented in Table 5.

Table 5 Comparison of the models for the prediction of an abnormal cardiac stress test

DISCUSSION

This study found statistically significant associations between abnormal CST and several factors,including male sex, CAD history, and hyperlipidemia, among patients with chest pain who presented to the hospital, had normal troponin tests, and completed a CST. The incidence rate of abnormal CST among low-risk patients was only 2.30%, while it was 12.19% among high-risk patients. NPV for all MLMs in this study did not reach the recommended value of 99%. However, the XGBoost MLM in this study provided a much better PPV of 24.33%.

Both the HEART pathway-ADP and EDACS-ADP have excellent sensitivity and NPV (99%), but the PPV (13.0% and 17.5%, respectively) is considerably low. This likely leads to unnecessary clinical intervention[7,8]. Despite the retrospective nature and limited explanatory variables in modeling in this study, the XGBoost MLM resulted in an increased PPV (24.33%). Patients who had CAD with the need for revascularization and who did not follow up within 30 d could not be identified in the HEART pathway-ADP and EDACS-ADP studies. Likewise, further clinical intervention was not completed for low-risk patients, who were subsequently discharged in the HEART pathway-ADP and EDACS-ADP studies. Therefore, no objective data exists to suggest that those discharged patients did not have CAD(such as non-obstructive CAD). Nevertheless, in our study, nuclear CST was part of the inclusion criteria to ensure there was objective data on CAD for all patients. Therefore, the inclusion criteria were stricter for our study.

In the HEART pathway-ADP and EDACS-ADP, ECG and all troponin tests (normal and abnormal)were predictor variables[7,8]. If a patient has elevated troponin values, then the resulting clinical decision is to recommend further clinical intervention the majority of the time. However, it is challenging to identify patients who have normal troponin values and need further clinical intervention.This study undertook the challenge of including only patients with normal troponin values, which is another reason that the selection criteria for this study cohort was restricted. However, it mirrors reallife clinical practice and the challenges that come with it. ECG was not included in this study and is a significant limitation. Since it was a retrospective study based on de-identified data, access to actual ECG data was not possible. Machine readings of the ECG are available. However, due to the poor PPV of machine readings for abnormal findings on ECG, machine readings were not included[18,19].Incorporating incorrect interpretations of ECG machine readings could have resulted in erroneous results and conclusions.

There has been increased interest in using MLMs and artificial intelligence in clinical decision-making in the past 8-10 years[12,14,15]. For patients who presented with chest pain to the ER, Stewartet al[12]conducted a systematic review of 23 studies that used MLMs to stratify these patients. There is significant heterogeneity in the studies included in this systematic review with differences noted in study design, type of MLM used, selected outcomes, comparisons made, data sharing, and validation.Despite the limitations, Stewartet al[12] reported that MLMs outperformed traditional risk stratification scores. Using high-sensitivity serum troponins, Doudesiset al[13] recently developed the MI3using gradient boosting MLM with sex, age, serial serum troponin concentrations, and the time interval between the serum troponin sampling as their parameters. This model also included both normal and abnormal troponin assays. MI3has reported a PPV of 70.4% for myocardial infarction if the MI3score was > 49.6 and an NPV of 99.8% if the MI3score was < 1.6. The specifics of their gradient boosting MLM have not been shared.

Zhanget al[14] conducted a study using MLMs for chest pain stratification, which shares some similarities in methodology to our study. The retrospective arm of their study was based on data from the electronic medical records. They excluded patients who had no follow-up data and did not include ECG for technical reasons (with no further explanation). The main differences in the study by Zhanget al[14] and our study included different outcomes (30-d AMI and all-cause mortalityvsabnormal CST,respectively), different number of explanatory variables (14vs10, respectively), troponin values (all valuesvsonly normal values, respectively), and the use of different MLMs.

Zhanget al[14] reported a robust sensitivity, specificity, and PPV of 92.90%, 88.50%, and 90.80%,respectively, for the 30-d risk of AMI during internal validation[14]. They observed similar sensitivity,specificity, and PPV for 30-d risk of all-cause mortality (77.50%, 99.99%, and 90.80%, respectively)[14].This represents one of the best models for risk stratification of patients who present to the ER with chest pain. In comparison, our study had a small sample size, used fewer explanatory variables, and included only normal troponins values, which likely led to lower predictive performance. Despite the robust performance of their MLM, Zhanget al[14] have not shared the MLM specifications in order to reproduce and externally validate their model. The heterogeneity of MLMs, lack of data sharing, and current lack of external validation are the major obstacles to widespread adoption of MLMs for risk stratification for patients presenting with chest pain.

Digitization of health records and exceptional computing power have enabled the use of artificial intelligence and MLMs in medicine. Taking advantage of these resources, our study created an MLM and is the first study to our knowledge that used an MLM for risk stratification of abnormal CST. Since abnormal CST can be used as a surrogate for the entire spectrum of CAD, this study promotes the idea that MLMs can be used to risk stratify for all CAD spectrums.

There are limitations to this study. First, ECG was not included in the study. ECG can improve both the PPV and NPV. Second, given the clear definitions of the computable phenotypes, the probability of missing a true risk factor or outcome is very low, but it is not zero. Third, we included only ten explanatory variables in this study, which is likely one of the reasons for lower NPV. Similar MLM studies with better predictive performance have incorporated 14 or more explanatory variables. Lastly,this study was a retrospective single-center study. Future studies replicating this study will strengthen the external validity of the current findings.

CONCLUSION

The current study achieved a better PPV for the entire spectrum of CAD (not just AMI) compared to the currently used risk stratification tools. These results highlight the potential of using MLMs in clinical decision-making. The results of this study also advanced the idea that a well-designed prospective study, which incorporates ECG and ensures proper follow-up, can achieve a much better PPV than currently used stratification tools (i.e.,the HEART pathway-ADP and the EDACS-ADP) while simultaneously maintaining an NPV of 99% for chest pain presentation to the ER. Data sharing and external validation of these prospective trials will be crucial to the recognition and adoption of MLMs.

ARTICLE HIGHLIGHTS

ACKNOWLEDGEMENTS

The University of Kansas Medical Center has an established i2b2-based Healthcare Enterprise Repository for Ontological Narration. The Healthcare Enterprise Repository for Ontological Narration is supported by the Clinical and Translational Science Award from the National Center for Advancing Translational Sciences, which has been awarded to the University of Kansas Clinical and Translational Science Institute. The authors are grateful to the Department of Clinical Informatics at the University of Kansas Medical Center for their help in providing de-identified data using the Healthcare Enterprise Repository for Ontological Narration.

FOOTNOTES

Author contributions:Shafiq M was involved in all aspects of this study, including but not limited to study design,data collection, data analyses, and writing of the abstract and manuscript; Mazzotti DR was involved in study design,data collection, and data analyses; Gibson CA assisted in writing the abstract and manuscript.

Institutional review board statement:Institutional Review Board approval was not required because the data was deidentified.

Informed consent statement:In accordance with the retrospective design of the study and de-identified data, no informed consent was required.

Conflict-of-interest statement:All the authors report no relevant conflicts of interest for this article.

Data sharing statement:All relevant data have been provided in this article. No additional data are available.

STROBE statement:The authors have read the STROBE Statement – a checklist of items, and the manuscript was prepared and revised according to the STROBE Statement – a checklist of items.

Open-Access:This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BYNC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is noncommercial. See: https://creativecommons.org/Licenses/by-nc/4.0/

Country/Territory of origin:United States

ORCID number:Muhammad Shafiq 0000-0002-0717-1764; Diego Robles Mazzotti 0000-0003-3924-9199; Cheryl Gibson 0000-0002-4025-8845.

S-Editor:Wang JJ

L-Editor:A

P-Editor:Wang JJ