The paper introduces calibrated concept profiles, a method that treats direct neuron readings as measurable quantities. By sampling prompts from a defined distribution and generating two independent batteries, the authors compute a per‑neuron profile expressed in the neuron’s own activation spread, complete with noise units and a minimum resolvable change. The approach is applied to OpenCoder‑1.5B across checkpoints spanning two trillion training tokens, providing unit‑scaled, chance‑level‑aware, and checkpoint‑specific construct attribution for each neuron.
By Piotr Wilam
The paper introduces Tokenization with States (TWS), a method that segments neural recordings into behaviorally meaningful states defined by task events and represents each state as a token based on population geometry, bypassing neuron- or session-specific embeddings. TWS successfully decodes movement from regime boundaries that carry no target information and transfers across species and recording modalities with only a linear probe. The approach demonstrates that tokens derived from shared population dynamics can generalize across sessions, outperforming per-neuron foundation models that fail to generalize.
By Sangyoon Bae, Jiook Cha
The paper investigates neural text degeneration by measuring the fixed‑point structure of short‑window argmax maps across 17 pretrained models, using 96 random two‑token starts without prompts. It finds a stable four‑way classification that varies across model families and scales, with some models funneling to a single endpoint token while others do not, and shows that this behavior is not solely determined by training data or corpus frequency. The study demonstrates that repetition phenomena are not uniformly explained by either training data or network architecture alone, highlighting the complexity of neural text generation dynamics.
By Nicol\'as Vera Z\'u\~niga
arXiv:2609.11149v3 Announce Type: replace-cross
Abstract: How fast does a language model degrade when trained on its own outputs? Theory traces it to gradually accumulating errors, while experiments...
By Yangze Liu, Zhongyi Han
The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.
By Matteo He, William F. Shen, Alex Iacob, Andrej Jovanovic, Xinchi Qiu, Nicholas D. Lane
The paper investigates whether groups of neurons that influence a model’s answer are uniquely necessary and transferable across inputs. By exhaustively testing all 4,096 subsets of 12 MLP neurons in Qwen3 and other models, the authors find that only a few neurons are often sufficient to alter predictions, that many subsets can independently flip answers, and that the effect is strongest on the same input chain but weaker on different chains. The study shows that causal neuron circuits are not uniquely defined and that their influence can vary with context and model architecture.
By \"Omer Tekin Yavuz (\.Ileti\c{s}im Yaz{\i}l{\i}m)
Delta N-gram introduces a forward‑only write path into a language model’s native hashed N‑gram table, preserving the pretrained table while adding evidence through a sparse patch Δ. The method uses a 4.1 M‑parameter writer that emits gated targets per event and applies a block‑averaged delta rule to keep each row a convex combination of its old value and the new targets. Experiments on a 0.6 B prototype and Qwen3.8‑Flash‑Next show that the delta writer maintains more document‑specific memory and reduces drift compared to online gradient writes, achieving stable performance over thousands of writes.
whyItMatters":"The approach allows continual updates to a language model’s memory without altering its pretrained parameters, enabling efficient, stable, and document‑specific knowledge retention during inference."
By Hao Wu
arXiv:2606. 13705v1 Announce Type: cross Abstract: Yes.
By Aristotelis Lazaridis, Aman Sharma, Dylan Bates, Brian King, Vincent Lu, Jack FitzGerald
arXiv:2607. 11958v1 Announce Type: new Abstract: Under the free energy principle, a predictive system does not observe reality directly; it maintains a generative model of the world and experiences that model's best current hypothesis.
By MD Ibrahim Hossain Ridoy
NeuronSpark is a self‑attention‑free language model that uses input‑dependent spiking membrane dynamics, allowing each neuron’s decay rate, gain, and firing threshold to adapt to the current token. This selective spiking approach enables billion‑parameter pretraining from random initialization without relying on backbone transfer, conversion, or distillation, achieving competitive zero‑shot performance and a fast–slow hierarchy. On a 1.1B‑parameter, 2.1B‑token training run, NeuronSpark attains a 40.0 eight‑task zero‑shot average, surpasses comparable‑budget Pythia/Mamba checkpoints, and demonstrates significant energy and cycle reductions on FPGA implementations.
By Zhengzheng Tang
arXiv:2607. 27539v1 Announce Type: new Abstract: Exact deletion from persistent language-model memory depends on how that memory represents a record.
By Vishwajith Ramesh
arXiv:2610. 11475v1 Announce Type: new Abstract: Population gradient flow is a common tool for reasoning about how neural networks adapt, including after pretraining.
By Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai, Jiaqi Wu, Chenyu Zhu, Tong Che