A multimodal generative AI copilot for human pathology

Computational pathology has significantly advanced with the development of task-specific predictive models and task-agnostic self-supervised vision encoders. However, the potential of generative AI in creating general-purpose multimodal AI assistants for pathology remains largely unexplored. This study addresses this gap by introducing PathChat, a vision-language AI assistant designed to interact with pathologists using both visual (histology images) and natural language inputs. The increasing accessibility of digital slide scanning, advancements in AI, availability of diverse datasets, and high-performance computing resources have fueled the progress in computational pathology, enabling deep learning applications in cancer subtyping, grading, metastasis detection, survival prediction, tumor origin prediction, and mutation prediction. While significant progress has been made in vision-only models, the crucial role of natural language in pathology – as a key to unlocking medical knowledge, a supervisory signal for model development, and a medium for user interaction – has not been fully integrated. Large-scale vision-language representation learning has shown promise in general machine learning, enabling capabilities like zero-shot image recognition and text-to-image retrieval. In medical imaging, including computational pathology, researchers have started utilizing paired biomedical images and text data for visual-language model training. However, a general-purpose multimodal AI assistant that seamlessly integrates visual and natural language understanding for pathology is still needed. PathChat aims to fulfill this need by creating a tool that can significantly improve diagnostic accuracy, provide comprehensive explanations, and enhance the overall workflow for pathologists.

The paper reviews existing literature on computational pathology, highlighting the progress made in developing task-specific predictive models and task-agnostic self-supervised vision encoders. It also notes the limited exploration of generative AI in creating general-purpose multimodal AI assistants for pathology. The authors discuss existing vision-language models and their applications in other medical imaging fields, emphasizing the need for a model specifically tailored to the complexities and nuances of pathology. The review underscores the crucial role of natural language in pathology and its potential for improving model development and user interaction. It also references several studies that have explored vision-language models in specific medical domains, including pathology and radiology, but highlights the lack of a truly general-purpose multimodal AI assistant for pathology.

PathChat was built by adapting a state-of-the-art (SOTA) vision-only, self-supervised pretrained encoder model (UNi) for pathology. This vision encoder was combined with a 13-billion-parameter pretrained Llama 2 large language model (LLM) through a multimodal projector module. The entire system was then fine-tuned on a large, curated dataset of over 456,000 diverse visual-language instructions comprising 999,202 question-and-answer turns. This dataset included various formats such as multi-turn conversations, multiple-choice questions, and short answers, drawn from diverse sources. The UNi encoder was initially pretrained on approximately 100 million image patches from around 100,000 slides using self-supervised learning. Further vision-language pretraining was performed on the UNi encoder using 1.18 million pathology image-caption pairs. This aligned the image representation space with that of pathology text. The resulting architecture was fine-tuned on the curated instruction-following dataset to create PathChat. For evaluation, PathChat was compared with several multimodal vision-language AI assistants and GPT-4. Two main evaluation strategies were used: a multiple-choice diagnostic question assessment, and an open-ended question assessment involving human pathologist ranking of responses. In the multiple-choice evaluation, PathChat's diagnostic accuracy was assessed on cases with diverse tissue types and disease models, using both image-only and image-with-clinical-context settings. The open-ended evaluation involved a panel of seven expert pathologists who ranked the responses generated by PathChat and other models for relevance, correctness, and succinctness. Pathologists were blinded to the model generating each response. Head-to-head comparisons were performed to determine win/loss rates.

PathChat achieved state-of-the-art performance on multiple-choice diagnostic questions from cases with diverse tissue types and disease models, outperforming other models such as LaVA-1.5 and LaVA-1.6. The addition of clinical context consistently improved PathChat’s accuracy, highlighting its ability to leverage multimodal information. In the open-ended question evaluation, PathChat generated more preferable and higher-ranked responses than all other LLMs tested. Compared to GPT-4, PathChat showed a favorable median win rate of 56.5% across seven independent pathologists. PathChat exhibited superior performance in microscopy and diagnosis categories, which require detailed image analysis. While GPT-4 performed better on clinical and ancillary testing questions (which often didn’t require image analysis), PathChat demonstrated significant improvement (+26.4% compared to LLMIVA-L Med and +48.9% compared to GPT-4) in overall accuracy for open-ended questions where pathologist consensus was reached (235 out of 260). In summary, PathChat demonstrated superior performance in both multiple-choice and open-ended question evaluations, surpassing existing models in accuracy and the quality of generated responses, particularly when analyzing images.

PathChat’s superior performance demonstrates the potential of multimodal generative AI in revolutionizing human pathology. Its ability to integrate visual and textual information, coupled with advanced LLM capabilities, enables accurate diagnoses and comprehensive explanations. The results address the research question by showcasing a robust AI copilot capable of assisting pathologists. The study's significance lies in its demonstration that a specialized, fine-tuned model can outperform general-purpose AI models in a complex medical domain. PathChat's ability to handle diverse query types and seamlessly integrate clinical context broadens its applicability across various pathology tasks. The findings are highly relevant to the field, suggesting a potential paradigm shift in diagnostic workflows, research methodologies, and educational tools. The ability to combine visual features with clinical context and medical knowledge suggests wider applications in diagnostics and beyond.

This study successfully demonstrates the feasibility and advantages of a multimodal generative AI copilot for human pathology. PathChat significantly outperforms existing models in both diagnostic accuracy and the quality of responses, offering a valuable tool for pathologists. Future research should focus on enhancing PathChat's capabilities by incorporating whole-slide image analysis, addressing limitations related to outdated information, and integrating it with existing pathology tools. Continued development and refinement of such AI copilots could revolutionize pathology practice, research, and education.

While PathChat demonstrates significant advancements, limitations exist. The model was trained on a retrospective dataset, potentially reflecting past scientific consensus rather than current knowledge. Handling whole-slide images (WSIs) instead of selected regions of interest (ROIs) could enhance its capabilities. Additionally, the reliance on human expert ranking for open-ended question evaluation introduces subjectivity. Finally, real-world deployment and validation are crucial to ensure consistent and reliable performance in diverse clinical settings.