← Back to all news
arXiv Computation and Language By Karolina Dro\.zd\.z, Micha Heilbron

Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives

Read the original on arXiv Computation and Language →

The study investigates how language models track entities across discourse, comparing them to human performance using naturalistic narratives. It finds that humans experience a decline in entity tracking only with increased narrative complexity, not length. Remarkably, language models with as few as 410 million parameters already match human-level tracking, and larger models surpass human performance, indicating that this core linguistic skill emerges at smaller scales than previously believed.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Sep 11

Characterizing Narrative Content in Web-scale LLM Pretraining Data

The paper presents a detailed examination of narrative elements—agency, setting, and events—within the Dolma web-scale pretraining corpus. Using a framework of 11 interpretable dimensions, the authors hand‑annotated 400 passages, expanded this to a 25,000‑passage LLM‑labeled dataset, and trained NarraBERT models to predict narrative features across 13 million passages, producing the NarraDolma dataset. The study reveals that narrative structure is measurable at scale and that narrative qualities vary unevenly across different data sources, topics, and formats, highlighting gaps in current data curation practices.

By Teagan Johnson, Elliott Ash, Andrew Piper, Maria Antoniak
llms
More like this →