The paper investigates energy-based transformers as predictors of reading difficulty, extending the use of transformer language models in psycholinguistics. It demonstrates that the energy measure from these models robustly predicts reading times across multiple corpora, outperforming traditional metrics like surprisal and attention entropy. In a controlled experiment on relative clause processing, energy captures known asymmetries, suggesting it may unify previously complementary predictors.
arXiv:2608.23818v2 Announce Type: replace Abstract: Transformer-based language models are widely used as models of human language processing, yet their attention mechanisms allow lossless access to t...
The study evaluates eleven autoregressive transformer models on English agreement attraction scenarios using a surprisal-based approach. Results show that while transformers match human reading times for prepositional phrase configurations, they perform poorly on object‑extracted relative clauses, with predictions diverging across models and failing to capture human interference patterns. The authors argue that current transformers cannot adequately model human morphosyntactic processing and call for more rigorous, comprehensive testing to avoid misleading conclusions from limited syntactic setups.
STRUCTURALCOST is a self‑paced reading dataset comprising 475 participants and 40,800 observations that isolates the processing cost of long‑distance subject‑verb dependency resolution. The study replicates a known psycholinguistic finding at NLP scale: human reading times at the main verb increase with dependency length, driven by syntactic embedding beyond linear distance. Various language models—n‑gram, SSMs, and transformers—partially mirror this graded difficulty profile but consistently underestimate the integration cost humans incur, a gap that persists across architectures and model sizes.
The study examines whether Dutch language models exhibit coherence‑illusion effects similar to human readers, using texts that refer back to earlier context with words like ‘again’ and ‘too’. Surprisal at the critical word aligns with human acceptability and eye‑tracking data, showing that models are more surprised by incoherent continuations unless a matching distractor is present. Attention entropy and an energy metric from associative‑memory literature reveal heads that behave differently under coherence versus incoherence, and ablating these heads demonstrates transfer effects across experiments, indicating a shared underlying mechanism.
arXiv:2510. 05969v3 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly deployed on complex reasoning tasks, yet little is known about their ability to internally evaluate problem difficulty, which is an essential capability for adaptive reasoning and efficient resource allocation.
arXiv:2505.12196v2 Announce Type: replace Abstract: The impressive linguistic abilities of large language models (LLMs) have recommended them as models of human sentence processing, with some conject...
The study introduces a latent‑process multinomial processing tree (MPT) model to analyze human reading and comprehension of garden‑path sentences across four reading paradigms (eye tracking, uni‑ and bidirectional self‑paced reading, Maze). The model separates the likelihood of an incorrect initial analysis, the cost of encountering an incompatible continuation, and the cost of syntactic reanalysis, yielding more realistic parameter estimates when inattentive trials are considered. Cross‑validation shows that this MPT model predicts human reading patterns and end‑of‑trial judgments better than a model relying solely on large‑language‑model (LLM) surprisal, and that incorporating surprisal as an additional predictor further improves fit.
The paper investigates attention sinks—tokens or positions that attract disproportionate attention—in text-to-image transformers, comparing bidirectional diffusion models and causal autoregressive models. It finds that both model types exhibit attention sinks, but with distinct patterns: autoregressive models focus on the first and global tokens, while diffusion models develop sinks near modality boundaries, within text regions, and at spatial corners. Perturbation experiments show that sink-related heads influence generation more than non-sink heads, and introducing register tokens can redistribute attention without harming performance.
The paper presents a mean‑field analysis of attention in language models, defining an average attention kernel that propagates representations layer by layer. When conditioned on a whole corpus, the kernel predicts the average evolution of representation geometry; when conditioned on a single context, it predicts the expected geometry for that context. The difference between actual attention and the mean‑field prediction—called the mean‑field deviation—captures context‑specific computation, revealing how models diverge from average behavior during training and in few‑shot tasks.
The study examines how varying the context window size of GPT‑2 affects its alignment with human reading-time data. A U‑shaped relationship is found: very short windows (<20 tokens) capture local memory limits, while larger windows (500–1,000 tokens) provide the best overall fit. Counterfactual experiments that disrupt cross‑sentential coreference chains show that these long‑range coreference relations are crucial for the improved fit of larger context windows.
arXiv:2608. 04678v1 Announce Type: cross Abstract: Papers 1-2 of the Kathleen series showed that a byte-level, attention-free architecture built from a wavetable encoder and multi-scale reverberant state can match strong baselines on classification at ~450-700K parameters, without pretraining.
arXiv:2607. 08152v1 Announce Type: cross Abstract: On the recent EyeBench benchmark, predicting reading comprehension from eye movements exposes a stark gap: text-aware models using pretrained language models reach 56--63% AUROC, while gaze-only models operate at chance.