The study investigates representational drift in the OpenCoder‑1.5B language model by tracking how neurons encode 169 Python constructs during pretraining on two trillion tokens. While the loss for 16 grammar‑fixing constructs quickly reaches a floor, the specific neurons that carry these constructs continue to change throughout training, with no stable carrier set emerging even at the final checkpoint. The authors find that removing the identified carriers incurs a measurable performance penalty, yet the carrier sets remain dynamic, challenging the hypothesis that they settle over time.
By Piotr Wilam
The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.
By Matteo He, William F. Shen, Alex Iacob, Andrej Jovanovic, Xinchi Qiu, Nicholas D. Lane
The paper demonstrates that Mixture-of-Experts (MoE) models can avoid the need for calibration data when selecting which expert channels to load during decoding. Instead of using a calibrated threshold, a top‑k selection strategy—choosing the k highest‑scoring channels per token—achieves comparable or better performance without calibration. Additionally, the authors propose a reader that uses a ternary sketch and low‑rank mapping to recover skipped channels, achieving 98.1% of full‑model accuracy while reading only 17.8% of expert weights and speeding up inference by up to 2.69× compared to optimized vLLM.
By Geunhyoung Kim, Sung Ju Hwang
The paper shows that AI‑text detectors, rather than learning a clear AI‑versus‑human boundary, amplify an inherited predictability axis that already exists in language models. This amplification causes detectors to over‑flag fluent, formal human writing while missing high‑temperature AI outputs, and the bias persists across languages, code, and detector architectures. A training‑free operator can relocate the bias but cannot erase it, underscoring that the unfairness is a structural cost of out‑of‑distribution generalization.
By Alexander Smirnov
The paper demonstrates that Mixture-of-Experts (MoE) decoding can avoid the need for calibration data by using a top‑k channel selector instead of a calibrated threshold. Top‑k selection consistently matches or outperforms threshold‑based methods across various read budgets and tasks, eliminating issues like repetition‑loop collapse. Additionally, the authors propose a reader that uses a ternary sketch and low‑rank mapping to recover skipped channels, achieving 98.1% of full‑model accuracy while reading only 17.8% of routed‑expert weights and speeding up MoE blocks by up to 2.69× compared to optimized vLLM.
arXiv:2609.36426v1 Announce Type: cross
Abstract: A detector pretrained on a broad corpus is fine-tuned on a narrow domain, its in-domain accuracy improves, and it ships. We ask what happens meanwhil...
By Trung Minh Bui, Jongsul Moon, YoungOuk Kim, Jung-Hoon Hwang, Dongin Shin
The paper investigates how many transformer components influence a token prediction by measuring the absolute contribution of each unit and channel to the logit. It finds that thousands of components contribute to a single prediction, yet a small subset—often just dozens—carries the majority of the predictive mass. Across models ranging from 124 M to 7 B parameters, the proportion of the model involved in a prediction remains around one to three percent, independent of size, and the study demonstrates that specific components can be directly read and written to modify model behavior without additional training.
By Mark Oskin
arXiv:2608. 10986v1 Announce Type: cross Abstract: A growing class of methods probes a language model by feeding it its own output: self-consistency, iterated refinement, agentic loops.
By Nicol\'as Vera Z\'u\~niga
The paper investigates neural text degeneration by measuring the fixed‑point structure of short‑window argmax maps across 17 pretrained models, using 96 random two‑token starts without prompts. It finds a stable four‑way classification that varies across model families and scales, with some models funneling to a single endpoint token while others do not, and shows that this behavior is not solely determined by training data or corpus frequency. The study demonstrates that repetition phenomena are not uniformly explained by either training data or network architecture alone, highlighting the complexity of neural text generation dynamics.
By Nicol\'as Vera Z\'u\~niga
The paper introduces Concept-Targeted Attribution (CTA), a method that trains attribution graphs to explain the emergence of internal concept representations in language models, rather than just the final token prediction. CTA produces probe-specific circuits that reveal which internal computations drive a linear probe’s accuracy, and cross-layer transcoders demonstrate that these graphs contain predictive structure across multiple concept categories. Causal ablations show that probe-targeted and logit-targeted graphs capture distinct mechanisms, with probe-relevant features affecting internal concept scores and logit-relevant features altering generated tokens.
By Vedant Palit, Florent Draye, Terry Jingchen Zhang, Bernhard Sch\"olkopf, Zhijing Jin
arXiv:2607. 06925v1 Announce Type: new Abstract: Compact world models that condition on a language goal promise to ground relations such as ``put the red block left of the blue block'' using a sparse set of explicit \emph{reference anchors}.
By Yufeng Wang, Lu Wei, Haibin Ling
arXiv:2609.14754v1 Announce Type: cross
Abstract: Causal claims about large language model (LLM) internals rest on measurements. Those might include a projection, a cosine, an ablation delta, or an i...
By Orion Reblitz-Richardson