A multimodal generative AI copilot for human pathology

Computational pathology has rapidly advanced with both task-specific predictive models and task-agnostic self-supervised vision encoders. Yet, despite growth in generative AI, few general-purpose multimodal AI assistants tailored to pathology exist. We present PathChat, a vision–language AI assistant for human pathology built by adapting a domain-specific foundational vision encoder, coupling it with a pretrained large language model, and fine-tuning on more than 456,000 diverse visual–language instructions totaling 999,202 turns. PathChat is compared to several multimodal assistants and GPT-4. It achieves state-of-the-art performance on multiple-choice diagnostic questions across diverse tissues and diseases and generates explicit, readable, pathologist-preferred responses to open-ended pathology queries. As an interactive copilot capable of reasoning over visual and natural language inputs, PathChat has potential applications in pathology education, research and in-loop clinical decision-making.