The study evaluates eleven autoregressive transformer models on English agreement attraction scenarios using a surprisal-based approach. Results show that while transformers match human reading times for prepositional phrase configurations, they perform poorly on object‑extracted relative clauses, with predictions diverging across models and failing to capture human interference patterns. The authors argue that current transformers cannot adequately model human morphosyntactic processing and call for more rigorous, comprehensive testing to avoid misleading conclusions from limited syntactic setups.
arXiv:2609.00416v1 Announce Type: new Abstract: Probing studies have established that syntactic information is decodable in early and middle transformer layers, but what happens to that information i...
The paper investigates why AI-generated text has become harder to understand, focusing on psycholinguistic features linked to human processing difficulty such as object relative clauses, long syntactic dependencies, and low-frequency n-grams. Using a new corpus of AI-generated text from diverse domains, the study finds that frontier and near-frontier language models employ these difficult features more frequently than paraphrases from round-trip translation and human writing, including more inanimate subjects with animate verbs, longer dependencies, stacked nouns, and reduced object relatives. The authors hypothesize that AI models prioritize information density for tasks like math, code, and reasoning, diverging from human communication goals, and demonstrate this by fine‑tuning a model with a length penalty, which reduces human‑like linguistic behavior.
arXiv:2601.19926v3 Announce Type: replace-cross Abstract: We present a systematic review of 337 articles evaluating the syntactic abilities of Transformer-based language models (TLMs), reporting on o...
arXiv:2608.23818v2 Announce Type: replace Abstract: Transformer-based language models are widely used as models of human language processing, yet their attention mechanisms allow lossless access to t...
STRUCTURALCOST is a self‑paced reading dataset comprising 475 participants and 40,800 observations that isolates the processing cost of long‑distance subject‑verb dependency resolution. The study replicates a known psycholinguistic finding at NLP scale: human reading times at the main verb increase with dependency length, driven by syntactic embedding beyond linear distance. Various language models—n‑gram, SSMs, and transformers—partially mirror this graded difficulty profile but consistently underestimate the integration cost humans incur, a gap that persists across architectures and model sizes.
The paper investigates how adjectival modifiers affect the semantic plausibility of events, using the Adept benchmark of 16,000 English sentence pairs that differ by a single adjective. Experiments show that sentence transformers, despite being conceptually suited to the task, underperform compared to models like RoBERTa. The authors provide an error analysis and discuss the implications of their findings for future work on balancing training and test data.
The paper investigates energy-based transformers as predictors of reading difficulty, extending the use of transformer language models in psycholinguistics. It demonstrates that the energy measure from these models robustly predicts reading times across multiple corpora, outperforming traditional metrics like surprisal and attention entropy. In a controlled experiment on relative clause processing, energy captures known asymmetries, suggesting it may unify previously complementary predictors.
arXiv:2601. 04098v2 Announce Type: replace-cross Abstract: Transformer language models systematically prefer tokens at specific input positions regardless of semantic relevance---a phenomenon known as positional bias.
The study investigates whether pretrained transformer models encode functional words—such as pronouns and adverbs—in a way that mirrors human usage. By comparing embeddings of nouns with those of their functional counterparts in both isolated and parallel sentences, the authors find that functional words occupy a central yet distinct position in embedding space and that parallel lexicalized and functional sentences reside in different subspaces. Experiments show that only a mixed training set of functional and lexicalized sentences reveals shared syntactic and semantic structure, whereas training on either type alone fails to capture this parallelism.
arXiv:2606. 30815v1 Announce Type: cross Abstract: Recent work suggests that transformer language models show a bias towards human languages over unnatural ("impossible") languages argued to be unacquirable by humans.
arXiv:2512. 22227v3 Announce Type: replace-cross Abstract: We investigate whether graded states of mind form spectrum-like structure in transformer representation spaces.
arXiv:2609.37497v1 Announce Type: new Abstract: Modern transformer models excel at capturing semantic relationships through sentence embeddings, yet their ability to perform pragmatic reasoning remai...