Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction

Deep learning (DL)-based predictive models from electronic health records (EHRs) deliver impressive performance in many clinical tasks but typically require large, annotated datasets. Inspired by bidirectional encoder representations from transformers (BERT) in natural language processing, we propose Med-BERT, a contextualized embedding model adapted to structured EHRs and pretrained on 28,490,650 patients. Fine-tuning experiments on two disease prediction tasks across two clinical databases show that Med-BERT substantially improves area under the receiver operating characteristic curve (AUC) by 1.21–6.14%. Med-BERT is particularly effective with small fine-tuning sets, achieving over 20% AUC boosts in some settings or matching performance of models trained on datasets up to ten times larger than those without Med-BERT. Using a large, widely adopted vocabulary (ICD-9 and ICD-10) and multi-institutional data, Med-BERT demonstrates cross-dataset generalizability. These results suggest Med-BERT can enable high-performing predictive models with limited local data, reduce data collection burden, and accelerate AI-enabled healthcare.