logo
ResearchBunny Logo
Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs

Medicine and Health

Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs

L. Wang, X. Chen, et al.

Discover how prompt engineering can enhance the reliability of Large Language Models in answering medical queries. Learn from a study conducted by Li Wang, Xi Chen, XiangWen Deng, Hao Wen, MingKe You, WeiZhi Liu, Qi Li, and Jian Li, highlighting the crucial role of effective prompting strategies in medical accuracy.

00:00
00:00
Playback language: English
Abstract
This study explores the impact of prompt engineering on the consistency and reliability of Large Language Models (LLMs) in answering medical questions. Different prompt styles were used to query various LLMs regarding their agreement with osteoarthritis (OA) guidelines. GPT-4-Web with ROT prompting showed the highest overall consistency (62.9%), particularly for strong recommendations (77.5%). Reliability varied significantly across models and prompts (Fleiss kappa ranged from -0.002 to 0.984). The study suggests that appropriate prompt engineering can improve the accuracy of LLM responses to medical questions.
Publisher
npj Digital Medicine
Published On
Jan 01, 2024
Authors
Li Wang, Xi Chen, XiangWen Deng, Hao Wen, MingKe You, WeiZhi Liu, Qi Li, Jian Li
Tags
prompt engineering
Large Language Models
osteoarthritis
consistency
reliability
medical questions
accuracy

Related Publications

Explore these studies to deepen your understanding of the subject.

Entrepreneurial universities and integrated sustainability for the knowledge-based economy: self-perception and some structural challenges in the Gulf region
Business

Entrepreneurial universities and integrated sustainability for the knowledge-based economy: self-perception and some structural challenges in the Gulf region

E. Zaidan, R. Momani, et al.

Efficacy of early PET-CT directed switch to carboplatin and paclitaxel based definitive chemoradiotherapy in patients with oesophageal cancer who have a poor early response to induction cisplatin and capecitabine in the UK: a multi-centre randomised controlled phase II trial
Medicine and Health

Efficacy of early PET-CT directed switch to carboplatin and paclitaxel based definitive chemoradiotherapy in patients with oesophageal cancer who have a poor early response to induction cisplatin and capecitabine in the UK: a multi-centre randomised controlled phase II trial

S. Mukherjee, C. N. Hurt, et al.

Mediating effect of fat mass, lean mass, blood pressure and insulin resistance on the associations of accelerometer-based sedentary time and physical activity with arterial stiffness, carotid IMT and carotid elasticity in 1574 adolescents
Health and Fitness

Mediating effect of fat mass, lean mass, blood pressure and insulin resistance on the associations of accelerometer-based sedentary time and physical activity with arterial stiffness, carotid IMT and carotid elasticity in 1574 adolescents

A. O. Agbaje

Population Pharmacokinetic and Exposure–Response Analysis of Finerenone: Insights Based on Phase IIb Data and Simulations to Support Dose Selection for Pivotal Trials in Type 2 Diabetes with Chronic Kidney Disease
Medicine and Health

Population Pharmacokinetic and Exposure–Response Analysis of Finerenone: Insights Based on Phase IIb Data and Simulations to Support Dose Selection for Pivotal Trials in Type 2 Diabetes with Chronic Kidney Disease

N. Snelder, R. Heinig, et al.

Listen, Learn & Level Up
Over 10,000 hours of research content in 25+ fields, available in 12+ languages.
No more digging through PDFs, just hit play and absorb the world's latest research in your language, on your time.
listen to research audio papers with researchbunny