Natural language processing system for rapid detection and intervention of mental health crisis chat messages

Patients experiencing mental health crises often seek help through messaging-based platforms, but may face long wait times due to limited triage capacity. The authors built and deployed a machine-learning-enabled system to improve response times to crisis messages in a large, national telehealth provider network. A two-stage NLP system with key word filtering followed by logistic regression was trained on 721 electronic medical record chat messages, of which 32% were potential crises (suicidal/homicidal ideation, domestic violence, or non-suicidal self-injury). Model performance was evaluated on a retrospective test set (4/1/21-4/1/22, N=481) and a prospective test set (10/1/22-10/31/22, N=102,471). In the retrospective test set, AUC was 0.82 (95% CI: 0.78-0.86), sensitivity 0.99 (95% CI: 0.96-1.00), and PPV 0.35 (95% CI: 0.309-0.4). In the prospective test set, AUC was 0.98 (95% CI: 0.966-0.984), sensitivity 0.98 (95% CI: 0.96-0.99), and PPV 0.66 (95% CI: 0.626-0.692). The daily median time from message receipt to crisis specialist triage ranged from 8 to 13 minutes, compared to 9 hours before deployment. The study demonstrates that an NLP-based model can reliably identify potential crisis chat messages in a telehealth setting and integrate into clinical workflows to facilitate rapid triage.

Akshay Swaminathan, Iván López, Rafael Antonio Garcia Mar, Tyler Heist, Tom McClintock, Kaitlin Caoili, Madeline Grace, Matthew Rubashkin, Michael N. Boggs, Jonathan H. Chen, Olivier Gevaert, David Mou, Matthew K. Nock