From Answers to Insights: Unveiling the Strengths and Limitations of ChatGPT and Biomedical Knowledge Graphs

Purpose: Large Language Models (LLMs) demonstrate strong performance across natural language processing tasks, but face limitations in biomedical contexts that can yield inaccurate or inconsistent answers. Biomedical Knowledge Graphs (BKGs) organize structured information and have become key resources for biomedical knowledge management. This study assesses and compares ChatGPT and existing BKGs in question answering, biomedical knowledge discovery, and reasoning. Methods: We evaluated ChatGPT on questions from the Yahoo! Answers “Alternative Medicine” subcategory and queried a BKG to retrieve corresponding knowledge, assessing responses manually. We further tested knowledge discovery by prompting ChatGPT to suggest potential drug/dietary supplement (DS) repurposing candidates for Alzheimer’s disease (AD), and compared against BKG-based link prediction using knowledge graph embeddings. We also examined each system’s ability to establish associations between proposed entities. Results: ChatGPT (GPT-4.0) outperformed GPT-3.5 and BKGs in providing existing information, while BKGs demonstrated higher reliability and accuracy. ChatGPT showed limitations in novel discovery and in reasoning to establish structured links between entities compared to BKGs. Conclusions: Integrating LLMs with BKGs could leverage complementary strengths, optimizing task performance while mitigating risks to advance biomedical knowledge and benefit individual well-being.

Yu Hou, Jeremy Yeung, Hua Xu, Chang Su, Fei Wang, Rui Zhang