Natural language processing system for rapid detection and intervention of mental health crisis chat messages

Patients experiencing mental health crises often seek help through messaging-based platforms, but may face long wait times due to limited triage capacity. The authors built and deployed a machine-learning-enabled system to improve response times to crisis messages in a large, national telehealth provider network. A two-stage NLP system with key word filtering followed by logistic regression was trained on 721 electronic medical record chat messages, of which 32% were potential crises (suicidal/homicidal ideation, domestic violence, or non-suicidal self-injury). Model performance was evaluated on a retrospective test set (4/1/21-4/1/22, N=481) and a prospective test set (10/1/22-10/31/22, N=102,471). In the retrospective test set, AUC was 0.82 (95% CI: 0.78-0.86), sensitivity 0.99 (95% CI: 0.96-1.00), and PPV 0.35 (95% CI: 0.309-0.4). In the prospective test set, AUC was 0.98 (95% CI: 0.966-0.984), sensitivity 0.98 (95% CI: 0.96-0.99), and PPV 0.66 (95% CI: 0.626-0.692). The daily median time from message receipt to crisis specialist triage ranged from 8 to 13 minutes, compared to 9 hours before deployment. The study demonstrates that an NLP-based model can reliably identify potential crisis chat messages in a telehealth setting and integrate into clinical workflows to facilitate rapid triage.