logo
ResearchBunny Logo
Shared functional specialization in transformer-based language models and the human brain

Computer Science

Shared functional specialization in transformer-based language models and the human brain

S. Kumar, T. R. Sumers, et al.

Discover groundbreaking insights into how transformer-based language models, like BERT, align with human brain activity in language processing. This research by Sreejan Kumar and colleagues reveals significant correlations between model computations and specific brain regions, suggesting shared computational principles that bridge machine learning and neuroscience.

00:00
00:00
Playback language: English
Abstract
This paper investigates the functional specialization in transformer-based language models (like BERT) and its correspondence to human brain activity during language processing. By analyzing the "transformations" (computations performed by attention heads) within the Transformer architecture, the researchers demonstrate that these transformations account for significant variance in brain activity across the cortical language network. Furthermore, they show a structured relationship between specific attention heads, their computations, and activity in particular brain regions. These findings suggest shared computational principles between the models and the human brain.
Publisher
Nature Communications
Published On
Jun 29, 2024
Authors
Sreejan Kumar, Theodore R. Sumers, Takateru Yamakoshi, Ariel Goldstein, Uri Hasson, Kenneth A. Norman, Thomas L. Griffiths, Robert D. Hawkins, Samuel A. Nastase
Tags
transformer models
BERT
brain activity
language processing
computational principles
attention heads
cortical language network
Listen, Learn & Level Up
Over 10,000 hours of research content in 25+ fields, available in 12+ languages.
No more digging through PDFs, just hit play and absorb the world's latest research in your language, on your time.
listen to research audio papers with researchbunny