Computer Science

Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers

C. Si, D. Yang, et al.

This innovative study by Chenglei Si, Diyi Yang, and Tatsunori Hashimoto explores how large language models generate cutting-edge research ideas. Surprisingly, the ideas produced by LLMs were rated significantly more novel than those from human experts, despite some concerns regarding feasibility. Dive into the findings of this intriguing research!

00:00

Playback language: English

Index

Abstract

This paper investigates the ability of large language models (LLMs) to generate novel research ideas. A large-scale human study involving over 100 NLP researchers was conducted to compare LLM-generated ideas with those produced by human experts. The results show that LLM-generated ideas were judged as significantly more novel than human expert ideas (p < 0.05), although slightly weaker in terms of feasibility. The study also identifies limitations in current LLM capabilities, including a lack of diversity in idea generation and unreliable self-evaluation.

Publisher

Published On

Authors

Chenglei Si, Diyi Yang, Tatsunori Hashimoto

Related Publications

Explore these studies to deepen your understanding of the subject.

Interdisciplinary Studies

Does large-scale research infrastructure affect regional knowledge innovation, and how? A case study of the National Supercomputing Center in China

H. Yang, L. Liu, et al.

Physics

An fMRI study of scientists with a Ph.D. in physics confronted with naive ideas in science

G. Allaire-duquette, L. B. Foisy, et al.

Medicine and Health

The salivary metabolome of children and parental caregivers in a large-scale family environment study

J. A. Rothman, H. L. Piccerillo, et al.

Computer Science

AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays

S. Herbold, A. Hautli-janisz, et al.

Listen, Learn & Level Up

Over 10,000 hours of research content in 25+ fields, available in 12+ languages.

No more digging through PDFs, just hit play and absorb the world's latest research in your language, on your time.

listen to research audio papers with researchbunny