The Flow has not summarised this story yet — read it at arXiv Computation and Language.
STRUCTURALCOST is a self‑paced reading dataset comprising 475 participants and 40,800 observations that isolates the processing cost of long‑distance subject‑verb dependency resolution. The study replicates a known psycholinguistic finding at NLP scale: human reading times at the main verb increase with dependency length, driven by syntactic embedding beyond linear distance. Various language models—n‑gram, SSMs, and transformers—partially mirror this graded difficulty profile but consistently underestimate the integration cost humans incur, a gap that persists across architectures and model sizes.
The paper investigates energy-based transformers as predictors of reading difficulty, extending the use of transformer language models in psycholinguistics. It demonstrates that the energy measure from these models robustly predicts reading times across multiple corpora, outperforming traditional metrics like surprisal and attention entropy. In a controlled experiment on relative clause processing, energy captures known asymmetries, suggesting it may unify previously complementary predictors.
The study evaluates eleven autoregressive transformer models on English agreement attraction scenarios using a surprisal-based approach. Results show that while transformers match human reading times for prepositional phrase configurations, they perform poorly on object‑extracted relative clauses, with predictions diverging across models and failing to capture human interference patterns. The authors argue that current transformers cannot adequately model human morphosyntactic processing and call for more rigorous, comprehensive testing to avoid misleading conclusions from limited syntactic setups.
arXiv:2505.12196v2 Announce Type: replace Abstract: The impressive linguistic abilities of large language models (LLMs) have recommended them as models of human sentence processing, with some conject...
arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.
arXiv:2510.01030v2 Announce Type: replace Abstract: The human ability to translate diverse perceptual and linguistic inputs into structured behavior has been thought to rest on learning robust repres...