arXiv:2608. 12652v1 Announce Type: cross Abstract: Benchmark contamination is diagnosed today with n-gram overlap, with likelihood-based membership inference, or with canary strings, and each needs something usually unavailable: the training corpus, a well-chosen test statistic, or foresight at dataset release.
By Florian Braun
arXiv:2609.37824v1 Announce Type: cross
Abstract: Transformer language models build predictions through successive residual updates, but how their representations become specific to an eventual outco...
By Timur Mudarisov, Mikhail Burtsev, Radu State
arXiv:2607. 18348v1 Announce Type: cross Abstract: We propose a transition-centred geometric analysis of transformer residual streams.
By Sunit Bhattacharya, Ravi Shankar Kolli
arXiv:2609. 12591v1 Announce Type: new Abstract: Foundation models are increasingly adapted through fine-tuning, model editing, and alignment procedures while retaining previously acquired capabilities.
By Hendrik Droste, Christian Medeiros Adriano, Kathrin Korte, Holger Giese
arXiv:2607. 06639v1 Announce Type: cross Abstract: On modular arithmetic, a network's embedding keeps compressing for tens of thousands of steps after it has already generalized.
By Truong Xuan Khanh
The paper investigates how many transformer components influence a token prediction by measuring the absolute contribution of each unit and channel to the logit. It finds that thousands of components contribute to a single prediction, yet a small subset—often just dozens—carries the majority of the predictive mass. Across models ranging from 124 M to 7 B parameters, the proportion of the model involved in a prediction remains around one to three percent, independent of size, and the study demonstrates that specific components can be directly read and written to modify model behavior without additional training.
By Mark Oskin
The study investigates why Lund‑plane and Particle Transformer (ParT) jet taggers can perform differently by using model stitching to isolate representation from computation. By linking frozen early layers of one network to frozen late layers of another, the authors show that a known deficit in the primary Lund representation is purely representational, not computational, and that this gap is invisible in AUC but appears in background rejection at low signal efficiency. Rebuilding the representation on the full Cambridge/Aachen declustering tree largely closes the gap, revealing that the primary branch carries most of the discriminative power while secondary splittings mainly make the representation learnable.
By Saurabh Rai, Sanmay Ganguly
The paper critiques the common practice of evaluating depth usage in depth‑recurrent language models by truncating depth during inference and measuring performance decline. It argues that this method conflates three distinct effects—fewer block applications, reduced computation, and an out‑of‑distribution readout—yet is usually interpreted as measuring only the second. To address this, the authors introduce the Depth Control Protocol (DCP), a suite of positive and negative controls that isolate each factor, along with a training intervention to confirm causality, specifically tailored for depth‑wise weight‑sharing architectures.
By Ha Van Dau, Thanh Tung Khuat, Nguyen Thanh Dung
arXiv:2608. 08652v1 Announce Type: cross Abstract: We present \LegoLM{}, a structured weight-sharing compression framework for large language models grounded in a systematic study of why global weight sharing fails and how to fix it.
By Joseph Bingham
arXiv:2608.02091v2 Announce Type: replace
Abstract: A bfloat16 transformer can train normally, then collapse abruptly. Prior work links collapse to structured attention errors and shows QK normalizat...
By Shuxiao Xie, Shuyang Xie, Yuan Cao, Dezhi Ran, Wei Yang, Tao Xie
The study investigates whether open‑weight language models can introspect on their own internal computations. Using the Open‑Weight Masked Introspection (OWMI) framework, researchers intervened on various internal components of eight models and asked them to report whether changes had occurred. Across 78,000 measurements, none of the models reliably distinguished real interventions from sham ones, with AUROC values essentially at chance.
Why It Matters: The findings suggest that current open‑weight models lack the ability to audit their own internal states, highlighting a limitation for oversight that relies on a model’s self‑reporting.
By Emilio Ferrara
The paper introduces Lexical Reference Frames (LexFrame) as a method to interpret language model residual streams by treating each hidden state as having a native reference frame induced by subsequent layers. Unlike fixed lenses or point‑cloud approaches, LexFrame reads a state through the model’s remaining layers using a set number of attention rounds, achieving high coherence (0.97) and accurate state reconstruction (within 0.16 nats). The approach also provides faithful attribution, token‑level steering, and potential early‑exit savings, making every hidden state readable, attributable, and steerable.
By Jian Gu, Hongyu Zhang, Chunyang Chen, Aldeida Aleti