Quantifying the impact of AI recommendations with explanations on prescription decision making

This study investigates how AI recommendations, with or without explanations (XAI), influence physician prescription decisions in critical care. 86 ICU doctors participated in a modified between-subjects design, evaluating four arms: baseline, peer recommendations, AI suggestions, and AI suggestions with XAI (simple feature importance). Results showed a strong influence of AI (but not peer) recommendations on prescriptions, while simple XAI had minimal additional impact. Clinician attitudes towards AI and experience did not correlate with AI-influenced decisions, nor did self-reported XAI usefulness correlate with its influence on prescriptions. The findings highlight the limited marginal impact of simple XAI in this context and question the reliability of self-reported data as a metric for assessing XAI in clinical settings.

Myura Nagendran, Paul Festor, Matthieu Komorowski, Anthony C. Gordon, Aldo A. Faisal