The paper investigates how six naturalistic and synthetic input perturbations affect decoder‑only language models at three levels: output behavior, hidden‑state geometry, and attention‑head function. Using GPT‑2 and Qwen2.5 checkpoints, the authors analyze layerwise geometry with centered kernel alignment and intrinsic dimension, and examine attention‑head responses in GPT‑2. They find that perturbation types produce distinct metric profiles that are not fully captured by output measures and vary across checkpoints, highlighting the need for multi‑level evaluation of robustness.
By Dun Li Chan, Emily Liu, Niyathi Allu, Christian Hoang
The study investigates how lexical perturbations—such as keyboard noise, character swaps, and filler insertion—affect large language models (LLMs) on reasoning benchmarks. Four open-weight instruction-tuned models and frontier models were evaluated, revealing that character-level perturbations significantly reduce accuracy, especially on multi-step reasoning tasks, while filler insertion has minimal impact. The authors attribute this asymmetry to Attention Diversion, where fragmented subword tokenization draws disproportionate attention in middle and final transformer layers; they demonstrate that both token content and attention allocation are coupled, making it difficult for inference-time repair strategies to fully recover performance.
By Jiaqian Zhu, Yang Zhang, Junhua Ding, Xiaowei Yu
arXiv:2607. 20524v1 Announce Type: new Abstract: Mean cross-positional attention degradation is widely reported in transformer interpretability, yet whether it causally limits contextual retrieval remains untested.
By Sagar Dangal, Manoj Shakya
arXiv:2606. 07604v1 Announce Type: cross Abstract: Analyzing attention weights has become a standard approach for interpreting the information flow of Large Language Models (LLMs).
By Harry Jake Cunningham, Nicola Muca Cirone
The paper investigates how post‑softmax attention weights behave across different language models by marginalizing over query positions to create a joint token‑head space. Across more than 60 diverse LLMs, token‑wise marginal attention reveals a text‑intrinsic signal that is conserved across models, while head‑wise marginal attention yields a model‑private signature that is consistent across documents. This insight enables a practical method for estimating per‑head budgets for key‑value cache eviction, achieving competitive performance on standard benchmarks without per‑document recomputation or training.
By Valentino Maiorca, Walter Nelson, Francesco Locatello
arXiv:2607. 00852v1 Announce Type: cross Abstract: This work studies the hidden-state inversion problem: recovering the original input token sequence of a decoder-only language model from its last-layer hidden states.
By Miko{\l}aj S{\l}owikowski, Maciej Witold Majewski
arXiv:2606. 15521v1 Announce Type: cross Abstract: Tokenization introduces representational redundancy: under a fixed token vocabulary, every byte string admits many valid token encodings, or segmentations, that decode to the same surface string.
By Kanishk Jain, Matthew Day, Tankut Can
arXiv:2606. 05486v1 Announce Type: cross Abstract: Prompt ambiguity is a common source of failure in large language models, but is difficult to localize because it is a latent property of the prompt, while existing attribution methods are designed to explain observable outputs such as logits or generated tokens.
By Govind Ramesh, Yao Dou, Wei Xu
arXiv:2607. 03738v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) generate responses autoregressively, integrating visual and linguistic information in an evolving context.
By Varun Gupta, Vineet Gandhi, Makarand Tapaswi
The paper introduces a three-level evaluation framework—behavioral deployment, LM-head readout, and probe recoverability—to distinguish whether a language model fails a syntactic test by not encoding structure or by failing to use it. Using a trilingual control-dependency benchmark, the authors find that probe recoverability consistently exceeds LM-head readout, which in turn exceeds behavioral deployment across seven models and three languages, with the largest gap observed in Qwen3-0.6B Instruct. Layer-localized activation patching shows that instruction tuning shifts the decoded layer later, suggesting decoding favors surface shortcuts and that behavioral evaluation understates what models encode while probing alone overstates what they deploy.
By Zhenyan Lu, He Wang, Xiaohui Huang
arXiv:2607. 09204v1 Announce Type: cross Abstract: Pretrained language models often exhibit structured weight spectra, suggesting that training may repeatedly produce similar layerwise and component-wise organization.
By Konstantin Garbers, Nicholas Oh
arXiv:2606. 01060v1 Announce Type: cross Abstract: Preference alignment has substantially improved the observable behavior of large language models, yet it remains unclear what alignment changes internally.
By Partha Pratim Saha, Samarth Raina, Mayur Parvatikar, Amit Dhanda, Vinija Jain, Aman Chadha, Amitava Das