arXiv:2610.16837v1 Announce Type: cross
Abstract: Syntactic language models (SLMs) that generate sentences along with their syntactic tree structures have shown promise in both syntactic generalizati...
By Pengcheng Wang, Zhexuan Zhang, Chao Lou, Kewei Tu
The study evaluates eleven autoregressive transformer models on English agreement attraction scenarios using a surprisal-based approach. Results show that while transformers match human reading times for prepositional phrase configurations, they perform poorly on object‑extracted relative clauses, with predictions diverging across models and failing to capture human interference patterns. The authors argue that current transformers cannot adequately model human morphosyntactic processing and call for more rigorous, comprehensive testing to avoid misleading conclusions from limited syntactic setups.
By Titus von der Malsburg, Sebastian Pad\'o
The paper investigates the reliability of attention‑head ablation as a causal inference tool in language models. Using GPT‑2 small, the authors find that a natural post‑projection zeroing method is almost uncorrelated with a corrected pre‑projection ablation and yields a completely different set of top‑5 important heads. They also show that binary accuracy can mask effects near performance floors or ceilings, whereas gold‑token log‑probability provides a graded signal. By employing a discovery/held‑out split and 1,000 matched random‑head and layer‑matched‑head controls, the corrected per‑head effect ranking remains highly stable (Spearman ρ = 0.974) and the top‑5 heads significantly outperform both control distributions (Monte Carlo p = 0.001). However, evidence for task specificity is weak on GPT‑2, and replication on DistilGPT‑2 confirms the intervention‑semantic and matched‑control findings.
By Juli Huang
arXiv:2610.33299v1 Announce Type: new
Abstract: To understand the emergence of grammatical behavior in language models, we track circuits for subject-verb number agreement in pythia-1b and ten seeds...
By Qing Yao, Sasha Boguraev, Tiago Pimentel, Kyle Mahowald
arXiv:2608.28924v1 Announce Type: new
Abstract: Linguistic theory has long recognized cross-linguistic syntactic regularities, leading to claims that these similar structures are processed by similar...
By Sasha Boguraev, Toshiki Nakai, Kyle Mahowald, Julius Steuer
The paper introduces Controlled Evidence Aggregation (CEA), a method that manipulates the counts and order of two symbols in fixed prompts to probe a frozen Transformer’s preference between two answer tokens. Using Balanced Utility Causal Selection (BUC), the authors identify small sets of attention heads whose ablation removes a large portion of the count effect, and show that these same heads also influence performance on a natural language task. The study demonstrates that symbolic probes can uncover compact head sets with causal impact on both synthetic and real-world language decisions.
By Jipeng Li, Pramod Khargonekar, Yanning Shen