TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records

Deep learning transformer-based models using longitudinal electronic health records (EHRs) have shown strong performance in predicting clinical diseases and outcomes. Pretraining on large datasets can improve downstream performance via finetuning with limited data. This study presents TransformEHR, a generative encoder-decoder transformer model pretrained with a novel objective: predicting all diseases and outcomes at a patient’s next visit using prior visits. The encoder-decoder framework and pretraining objective together achieve new state-of-the-art performance across multiple clinical prediction tasks. Compared to prior models, TransformEHR improves area under the precision-recall curve by 2% (p < 0.001) for pancreatic cancer onset and by 24% (p = 0.007) for intentional self-harm among patients with post-traumatic stress disorder (PTSD). TransformEHR attains a positive predictive value (PPV) of 8.8% among the top 10% highest-risk PTSD patients for intentional self-harm, exceeding published cost-effectiveness thresholds for practical intervention. TransformEHR generalizes well and can be readily finetuned for tasks with limited data.

Zhichao Yang, Avijit Mitra, Weisong Liu, Dan Berlowitz, Hong Yu