This study investigates how AI recommendations, with or without explanations (XAI), influence physician prescription decisions in critical care. 86 ICU doctors participated in a modified between-subjects design, evaluating four arms: baseline, peer recommendations, AI suggestions, and AI suggestions with XAI (simple feature importance). Results showed a strong influence of AI (but not peer) recommendations on prescriptions, while simple XAI had minimal additional impact. Clinician attitudes towards AI and experience did not correlate with AI-influenced decisions, nor did self-reported XAI usefulness correlate with its influence on prescriptions. The findings highlight the limited marginal impact of simple XAI in this context and question the reliability of self-reported data as a metric for assessing XAI in clinical settings.

Myura Nagendran, Paul Festor, Matthieu Komorowski, Anthony C. Gordon, Aldo A. Faisal