arXiv:2610.08037v1 Announce Type: new
Abstract: Language models frequently generate outputs in unintended languages or scripts, a phenomenon known as off-target generation. While existing research ha...
By David Kletz, Sandra Mitrovi\'c, Ljiljana Dolami\'c, Fabio Rinaldi
The paper "What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks" analyzes 14,767 arXiv submissions from 2022 to 2026 that introduce or update evaluation resources for large language models. It systematically maps changes in target systems, domains, evaluation materials, conditions, and scoring mechanisms, revealing a growing emphasis on action, interaction, and professional applications. The study also notes uneven development in model participation, with LLM-based scoring increasing in both agent and non-agent groups, while model-generated materials do not show a comparable rise.
By Chao Wang (Independent Researcher)
arXiv:2606. 11166v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly described as performing at the level of human experts on knowledge economy tasks.
By George Perrett, Javae Elliott, Jennifer Hill, Marc Scott
The paper introduces a three-level evaluation framework—behavioral deployment, LM-head readout, and probe recoverability—to distinguish whether a language model fails a syntactic test by not encoding structure or by failing to use it. Using a trilingual control-dependency benchmark, the authors find that probe recoverability consistently exceeds LM-head readout, which in turn exceeds behavioral deployment across seven models and three languages, with the largest gap observed in Qwen3-0.6B Instruct. Layer-localized activation patching shows that instruction tuning shifts the decoded layer later, suggesting decoding favors surface shortcuts and that behavioral evaluation understates what models encode while probing alone overstates what they deploy.
By Zhenyan Lu, He Wang, Xiaohui Huang
arXiv:2609.07876v1 Announce Type: cross
Abstract: Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linea...
By Arjun Patrawala, Jiahai Feng, Erik Jones, Jacob Steinhardt
arXiv:2507.21831v2 Announce Type: replace-cross
Abstract: LLMs are seeing widespread use for task automation, including automated coding in the social sciences. However, even though researchers have...
By Andreas Reich, Claudia Thoms, Tobias Schrimpf