A multimodal generative AI copilot for human pathology

Computational pathology has rapidly advanced with both task-specific predictive models and task-agnostic self-supervised vision encoders. Yet, despite growth in generative AI, few general-purpose multimodal AI assistants tailored to pathology exist. We present PathChat, a vision–language AI assistant for human pathology built by adapting a domain-specific foundational vision encoder, coupling it with a pretrained large language model, and fine-tuning on more than 456,000 diverse visual–language instructions totaling 999,202 turns. PathChat is compared to several multimodal assistants and GPT-4. It achieves state-of-the-art performance on multiple-choice diagnostic questions across diverse tissues and diseases and generates explicit, readable, pathologist-preferred responses to open-ended pathology queries. As an interactive copilot capable of reasoning over visual and natural language inputs, PathChat has potential applications in pathology education, research and in-loop clinical decision-making.

Ming Y. Lu, Bowen Chen, Drew F. K. Williamson, Richard J. Chen, Melissa Zhao, Aaron K. Chow, Kenji Ikemura, Ahrong Kim, Dimitra Pouli, Ankush Patel, Amr Soliman, Chengkuan Chen, Tong Ding, Judy J. Wang, Georg Gerber, Ivy Liang, Long Phi Le, Anil V. Parwani, Luca L. Weishaupt, Faisal Mahmood