This paper investigates the ability of large language models (LLMs) to generate novel research ideas. A large-scale human study involving over 100 NLP researchers was conducted to compare LLM-generated ideas with those produced by human experts. The results show that LLM-generated ideas were judged as significantly more novel than human expert ideas (p < 0.05), although slightly weaker in terms of feasibility. The study also identifies limitations in current LLM capabilities, including a lack of diversity in idea generation and unreliable self-evaluation.

Chenglei Si, Diyi Yang, Tatsunori Hashimoto