Performance of mental health chatbot agents in detecting and managing suicidal ideation

Advances in artificial intelligence (AI) technologies sparked a rapid development of smartphone applications designed to help individuals experiencing mental health problems through an AI-powered chatbot agent. However, the safety of such agents when dealing with individuals experiencing a mental health crisis, including suicidal crisis, has not been evaluated. In this study, we assessed the ability of 29 AI-powered chatbot agents to respond to simulated suicidal risk scenarios. Application repositories were searched and the app descriptions screened in search of apps that claimed to be beneficial when experiencing mental distress and offered an AI-powered chatbot function. All agents were tested with a standardized set of prompts based on the Columbia-Suicide Severity Rating Scale designed to simulate increasing suicidal risk. We assessed the responses according to pre-defined criteria based on the ability to provide emergency contact information and other factors. None of the tested agents satisfied our initial criteria for an adequate response, 51.72% satisfied the relaxed criteria for a marginal response, while 48.28% were deemed inadequate. Common errors included the inability to provide emergency contact information and a lack of contextual understanding. These findings raise concerns about the deployment of AI-powered chatbots in sensitive health contexts without proper clinical validation.

W. Pichowicz, M. Kotas, P. Piotrowski