The paper investigates the reliability of attention‑head ablation as a causal inference tool in language models. Using GPT‑2 small, the authors find that a natural post‑projection zeroing method is almost uncorrelated with a corrected pre‑projection ablation and yields a completely different set of top‑5 important heads. They also show that binary accuracy can mask effects near performance floors or ceilings, whereas gold‑token log‑probability provides a graded signal. By employing a discovery/held‑out split and 1,000 matched random‑head and layer‑matched‑head controls, the corrected per‑head effect ranking remains highly stable (Spearman ρ = 0.974) and the top‑5 heads significantly outperform both control distributions (Monte Carlo p = 0.001). However, evidence for task specificity is weak on GPT‑2, and replication on DistilGPT‑2 confirms the intervention‑semantic and matched‑control findings.
By Juli Huang
The paper investigates how visual interventions—training‑free edits applied to multimodal large language models—affect hallucinations and prediction quality. By applying a visual‑residual edit across successive decoder layers and matching its RMS strength, the authors identify an Evidence Anchoring Boundary (EAB) where the intervention’s influence drops sharply; effects before the EAB are 13.78–26.10× larger than after it. They further compare counterfactual subtraction across different question types and introduce calibrated selective routing, which only alters predictions when calibration supports an intervention, improving attribute accuracy by 1.56–1.81 percentage points while avoiding the harms of universal subtraction.
By Qiji Zhou, Junyi Hu, Peiyu Wang, Yue Zhang
The paper investigates how large language models (LLMs) handle affective state reasoning, a type of implicit conversational state that persists across topic shifts. It introduces Self‑targeted Internal Attention Boosting (SIAB), a method that identifies and amplifies sparse affective attention heads using causal localization and the model’s own behavior, without relying on external labels. Experiments on ECC and GoEmotions datasets across multiple LLMs and prompting settings show that SIAB improves macro‑F1 scores by up to +10.3 points on ECC and +5.5 on GoEmotions, and further gains are achieved when combined with external supervision.
By Wenqiu Tang, Zhen Wan, Yihui Wang, Takahiro Komamizu, Ichiro Ide
arXiv:2608. 14509v1 Announce Type: new Abstract: Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt.
By Zhelun Wu
arXiv:2506.19549v2 Announce Type: replace-cross
Abstract: Identifying which tokens and activations truly influence model predictions is critical for both the efficiency and interpretability of large...
By Debabrata Mahapatra, Shubham Agarwal, Apoorv Saxena, Subrata Mitra
The study investigates how the position of evidence documents relative to a prompt affects long‑context question answering across different sequence mixers—Transformer, state‑space, and hybrid Mamba models. Results show that Transformers exhibit a beginning‑position advantage that state‑space models lack, while both families tend to favor evidence near the end. Additional experiments reveal that prompt design, corpus choice, and model capability influence performance, and that position bias remains decodable across architectures even when response curves differ.
By Aman Behera, Namit Solanki, Mehul Anand