A validated, real-time prediction model for favorable outcomes in hospitalized COVID-19 patients

The COVID-19 pandemic significantly strained healthcare systems globally. While numerous prognostic tools have been published, few have undergone prospective validation, and none have reported successful implementation in clinical practice. This research addresses the critical need for accurate, real-time prediction of patient outcomes to optimize resource allocation and improve clinical decision-making. The focus is on predicting favorable outcomes, which aids in safely discharging patients and freeing up hospital beds. Existing models predominantly focus on predicting adverse events, while this study aims to identify patients at low risk, thereby facilitating efficient patient flow and resource management. The model's development and implementation involved collaboration between data scientists, EHR programmers, clinical informatics specialists, frontline clinicians, and clinical leadership, ensuring practical relevance and usability.

At the time of the study, at least 20 peer-reviewed papers described prognostic COVID-19 models, mostly predicting adverse events such as severe pneumonia, intubation, ICU admission, and death. Most used multiple variables and predicted composite outcomes. A majority of these models were trained on data from China, with limited validation on external or held-out datasets. Notably, no model reported successful clinical implementation for predicting favorable outcomes.

The study employed a two-stage approach for model development and deployment. Stage 1 involved creating a 'blackbox' model using various machine learning algorithms (logistic regression, Random Forest, LightGBM, and an ensemble) to predict favorable outcomes (absence of adverse events such as death, ICU admission, significant oxygen support, or readmission within 30 days). The data consisted of 3345 retrospective COVID-19 patient admissions from March 3 to April 26, 2020, with data on demographics, lab values (neutrophils, lymphocytes, eosinophils, platelets, BUN, creatinine, D-dimer, ferritin, LDH, troponin, C-reactive protein), and vital signs (heart rate, respiratory rate, SpO2, temperature). A favorable outcome was defined as the absence of any adverse events. Data was split into training (60%), validation (20%), and hold-out (20%) sets. The LightGBM model performed best. Stage 2 involved creating a 'parsimonious' model using a logistic regression with a reduced set of features selected through conditional independence tests, aiming for simplicity and ease of EHR integration. The parsimonious model included 13 features after an ablation analysis. Quantile normalization and linearization of variables were performed to handle outliers and non-linear relationships. Hyperparameters were tuned using the validation set. The final parsimonious model was implemented into the EHR to generate predictions every 30 minutes. Predictions were color-coded (green, orange, red) based on risk thresholds determined using the hold-out set to ensure high positive predictive value for the green (low risk) category. Prospective validation was performed using data from May 15 to May 28, 2020 (445 patients, 474 admissions, 109,913 predictions) with a 96-hour follow-up period. Model performance was evaluated using AUROC, AUPRC, PPV, and sensitivity.

The LightGBM blackbox model achieved high AUROC and AUPRC in the retrospective hold-out set. The parsimonious logistic regression model achieved comparable performance. In retrospective validation, the model achieved high average precision (88.6%) and discrimination (95.1%). Prospective validation yielded an AUROC of 90.8% and an AUC of 86.8%, similar to retrospective performance. Using the green threshold, the real-time model had a positive predictive value of 93.3% and 67.8% sensitivity, successfully identifying a significant portion of low-risk patients for safe discharge. The model's performance showed that patients identified as low-risk had a high probability of favorable outcomes, while the higher-risk categories also showed improved prediction of favorable outcomes compared to the retrospective hold-out set. Analysis of the timing of the first 'green' prediction for patients discharged alive showed that the majority never received ICU care, and those who did had a longer stay before the first green prediction but similar remaining length of stay after. Post-implementation monitoring of EHR usage indicated that clinicians adopted the model scores into their clinical workflow.

The study successfully developed and implemented a real-time prediction model for favorable outcomes in hospitalized COVID-19 patients. This approach contrasts with most existing models that predict adverse events. The model's high accuracy and seamless integration into the EHR improved clinical decision-making by enabling timely identification of patients suitable for discharge or transitioning to lower levels of care. The model's explanation feature facilitated trust and understanding among clinicians. The results emphasize the importance of prospective validation in a dynamically evolving clinical context, as observed during the shift from retrospective to prospective evaluation. While limitations related to data access in real-time exist, the use of a parsimonious model effectively mitigates these issues. Despite limitations in variable representation, the selected variables aligned with established prognostic indicators, highlighting the model’s clinical relevance. Future research will involve a randomized controlled trial to assess the model’s impact on clinically important outcomes, such as length of stay.

This study demonstrates the successful development, validation, and implementation of a real-time prediction model for favorable outcomes in hospitalized COVID-19 patients. The model's high accuracy, interpretability, and seamless integration into the EHR resulted in improved clinical decision-making and efficient resource allocation. Future work will focus on a randomized controlled trial and broader dissemination of the model to other healthcare institutions.

The study is limited by the potential for data bias stemming from the retrospective and prospective nature of the study, and potential for changes in clinical practices over time. The real-time data access limitations in the EHR constrained model complexity; some potentially useful data (e.g., complete comorbidity information) were unavailable for real-time prediction. While the model accurately predicts favorable outcomes, it cannot discriminate between reasons for oxygen therapy (e.g., chronic vs. acute respiratory failure).