Narrative coherence in neural language models

Neural language models (NLMs) are often framed as next-token predictors, yet they exhibit surprisingly human-like linguistic behaviors. This motivates applying psychological theory to probe their generative capacities. The study investigates whether NLMs display a core of coherent integration in language production, metaphorically akin to a human speaker's personal identity, by analyzing autobiographical narrative coherence using the Narrative Coherence Coding Scheme (NaCCS). Researchers elicited 2,010 autobiographical stories from GPT-3.5 and 2,010 from GPT-4, varying role attributes such as gender, mood, and age. A pilot set of 192 stories was rated by professional psychologists to build a fine-tuning dataset for an automatic NaCCS evaluator (fine-tuned GPT-3.5), enabling large-scale automated scoring. Across 4,020 stories, both models achieved narrative coherence levels comparable to human subjects, with GPT-4 slightly higher. Findings suggest substantial knowledge unification in the models, comparable to the integrative functions of a human self, while acknowledging anthropomorphism and interpretive caution.

Alessandro Acciai, Lucia Guerrisi, Pietro Perconti, Alessio Plebe, Rossella Suriano, Andrea Velardi