Machine learning (ML) and artificial intelligence (AI) are increasingly used in healthcare, but their impact in diverse contexts isn't well understood. Studies show that inaccurate AI recommendations can worsen clinical decisions, and that even experts make similar errors when presented with incorrect AI advice. This is especially concerning given that ML models often exhibit biases against minority groups. Large language models, readily deployable and powerful, show problematic prejudices, like associating Muslims with violence. Even models trained on clinical data favor majority groups in many prediction tasks. While bias in these models is established, its impact on real-world healthcare decision-making is unclear, particularly in crucial areas like mental health triage. This study aims to evaluate the impact of biased AI on decisions in mental health emergencies, focusing on whether biased models induce bias in human decisions and whether the style of AI recommendations influences adherence to those recommendations.

Existing research highlights the susceptibility of human decision-making to biased AI recommendations, particularly in high-stakes scenarios. Studies have shown that even with expert clinicians, inaccurate AI advice can lead to suboptimal treatment choices. The issue extends beyond simple errors, as AI models often demonstrate inherent biases reflecting societal prejudices against minority groups. This bias has been observed across various domains, from criminal justice to healthcare. While previous work identified the existence of these biases in AI models, there is a lack of understanding regarding their effect on real-world decision-making processes in healthcare settings. This study builds on this body of research by focusing specifically on the impact of biased AI in mental health emergency situations, where the potential consequences of biased decisions can be particularly significant.

A web-based experiment was conducted involving 954 participants (438 clinicians, 516 non-experts) recruited via email and social media from May 2021 to December 2021. Participants reviewed eight call summaries describing male individuals experiencing mental health emergencies. Each summary randomly assigned race (Caucasian or African-American) and religion (Muslim or non-Muslim). Participants then decided between calling for medical help or police assistance, based on perceived risk of violence. A baseline group made decisions without AI input. Other groups received AI recommendations, generated by a biased or unbiased GPT-2 model (fine-tuned on a custom dataset to introduce bias). The style of the recommendation was varied: prescriptive (e.g., "call the police") or descriptive (e.g., "flag for risk of violence"). Data was analyzed using logistic mixed-effects models to assess the impact of race, religion, AI recommendations, and recommendation style on participants' decisions. The model controlled for factors like participant demographics and attitudes towards policing. A detailed breakdown of participant demographics is provided in supplementary tables.

The study revealed that participants showed no inherent bias in their baseline decisions without AI advice. However, when presented with biased prescriptive AI recommendations, both clinicians and non-experts demonstrated significantly increased likelihood of calling the police for African-American and Muslim individuals. The odds ratios, with 95% confidence intervals, showcased a considerable disparity in decision-making influenced by biased AI. Conversely, when the same biased AI model provided descriptive flags instead of prescriptive recommendations, participants largely retained their unbiased decision-making. This demonstrates that the framing of AI advice is crucial; while prescriptive recommendations led to algorithmic adherence and resulting bias, descriptive flags enabled participants to overcome model bias and maintain fairer decisions. The difference in adherence to biased prescriptive and descriptive recommendations was statistically significant, indicating the importance of presentation style. Clinicians, while adhering to unbiased descriptive recommendations, largely ignored biased ones, suggesting a capacity to correct for model flaws with this presentation style.

This study's findings underscore the potential for biased AI to exacerbate existing societal biases in high-stakes decision-making. The ease with which the GPT-2 model was biased through fine-tuning highlights a critical risk in common ML workflows. The study's emphasis on recommendation style highlights the crucial role of human-AI interaction design. Prescriptive recommendations create blind adherence, while descriptive flags allow for human judgment to override potential biases. The results challenge the assumption that expert clinicians are immune to the influence of biased AI; even with experience, clinicians can be misled by biased prescriptive advice. Future ethical deployments of AI in healthcare require careful consideration of model auditing, recommendation style, and approaches like peer review to mitigate bias and improve decision-making.

This research demonstrates that even unbiased decision-makers can be influenced by biased AI recommendations, especially when presented prescriptively. Descriptive flags, in contrast, allow for human judgment to override algorithmic biases. This highlights the critical need for thorough model validation, thoughtful design of AI decision support systems, and continuous monitoring for bias. Future research should explore alternative methods of presenting AI information and techniques to further mitigate the influence of model bias in high-stakes settings.

The study's use of explicit racial and religious identifiers in call summaries might not fully capture the subtlety of real-world biases. A more nuanced approach, using implicit cues like names or accents, could potentially elicit stronger bias. The study's focus on mental health emergencies may limit the generalizability of findings to other healthcare contexts. Further research is needed to explore these areas and to investigate the effect of various factors on the mitigation of AI bias in different healthcare scenarios.