arXiv Machine Learning
High-dimensional layer-state trajectories reveal the internal dynamics of transformer inference
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jul 8
Transformers converge to invariant algorithmic cores
arXiv:2602. 22600v2 Announce Type: replace-cross Abstract: Training selects for behavior, not circuitry: many weight configurations can implement the same function.
arXiv AI
Oct 12
NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling
NeuronSpark is a self‑attention‑free language model that uses input‑dependent spiking membrane dynamics, allowing each neuron’s decay rate, gain, and firing threshold to adapt to the current token. This selective spiking approach enables billion‑parameter pretraining from random initialization without relying on backbone transfer, conversion, or distillation, achieving competitive zero‑shot performance and a fast–slow hierarchy. On a 1.1B‑parameter, 2.1B‑token training run, NeuronSpark attains a 40.0 eight‑task zero‑shot average, surpasses comparable‑budget Pythia/Mamba checkpoints, and demonstrates significant energy and cycle reductions on FPGA implementations.
arXiv Machine Learning
Jun 2
Task Structure Reverses Layerwise State Encoding in Sequence Models
arXiv:2606. 00926v1 Announce Type: new Abstract: Mechanistic studies of sequence models often treat layerwise state encodings as architectural traits: recurrent models concentrate readable state, attention-based models distribute it.
arXiv Machine Learning
Jun 9
Trajectory Geometry of Transformer Representations Across Layers
arXiv:2606. 09287v1 Announce Type: new Abstract: Understanding how transformer representations evolve across layers, not merely what they encode, remains an open problem in mechanistic interpretability.
arXiv Machine Learning
Jul 24
Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts
arXiv:2607. 20519v1 Announce Type: new Abstract: Looped Transformers increase test-time computation by repeatedly applying a shared recurrent block.
arXiv AI
Sep 1
Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models
The study investigates how post‑training of large autoregressive language models (ARMs) into masked diffusion models (MDMs) affects their internal computation. Across two 7B ARM‑MDM families and four diagnostic tasks, the authors find that MDMs retain much of the ARM’s high‑attribution pathways on prefix‑dominant tasks, but reorganize computation toward earlier layers on globally constrained tasks. Component‑level probes reveal that ARMs depend on sharply specialized components, whereas MDMs show weaker specialization and more diffuse output‑space alignment.