← Back to all news
arXiv AI By Melanie Subbiah, Chia-En Jacklyn Tsai, Kathleen McKeown

ToMStories: Interpreting Theory-of-Mind in Synthetic Narratives

Read the original on arXiv AI →

The paper introduces ToMStories, a dataset of about 6,000 synthetic narrative vignettes designed to test language models’ ability to infer characters’ psychological states such as emotions, fears, motivations, and wounds. Each vignette is paired with either a multiple-choice or open-ended question, and the states are presented implicitly, explicitly, or with conflicting cues. The authors evaluate thirteen open-weight models ranging from 1B to 72B parameters, finding that while the largest models perform on par with humans on multiple-choice tasks, they lag on open-ended questions, revealing simplifications in how models interpret and generate implicit psychological states.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Oct 12

Language Models Have Interpretable Narrative Mechanisms

The paper explores whether language models encode components that represent narrativity. By examining GPT‑2‑Small, Gemma‑2‑2B, and Llama‑3.1‑8B, the authors identify specific attention heads, MLP neurons, and sparse autoencoder features whose activations correlate with narrative properties. Ablating these components raises next‑token prediction loss more for high‑narrativity texts and diminishes narrativity in open‑ended generations, with the strongest effects seen in story completion and narrative question answering tasks.

By David Yiyang Liu, Aditya Joshi, Paul Dawson, Andrew Piper
llmsnlpbenchmarkssafety
More like this →