The paper introduces BRo-JEPA, a world model that learns modular arithmetic operations as rotations in latent space. Using MNIST and EMNIST datasets, BRo-JEPA achieves near-perfect zero‑shot generalization to unseen operations, outperforming standard supervised and JEPA baselines by a large margin. The model demonstrates that latent transformations can encode underlying algebraic structures, enabling strict zero‑shot operation generalization.
By Divyansh Jha, Yuanfang Xie, Brennen Yu, Varan Mehra
The paper investigates how transformer language models perform few‑shot learning for a simple addition task, showing that the ability is concentrated in a handful of attention heads. Using dimensionality reduction, the authors identify low‑dimensional subspaces—three heads with six‑dimensional spaces in Llama‑3‑8B‑Instruct—where specific dimensions encode the units digit via trigonometric patterns and magnitude via low‑frequency components. They also derive a mathematical identity linking aggregator and extractor subspaces, enabling tracking of information flow from examples to the final prediction.
By Xinyan Hu, Kayo Yin, Michael I. Jordan, Jacob Steinhardt, Lijie Chen
arXiv:2607. 07066v1 Announce Type: cross Abstract: Transformers have demonstrated a remarkable ability to learn algorithmic reasoning, yet mechanistic analyses have mostly focused on globally invertible operations such as cyclic addition and group composition.
By Zitong Andrew Chen, Junaid Hasan, Akhil Srinivasan, Hemkesh Bandi, Jarod Alper
arXiv:2606. 14108v1 Announce Type: cross Abstract: We introduce Adelic operation-preserved embeddings (AOE), a training-free representation that captures both a number's real value and its modular (p-adic) signatures.
By Suhyun Bae, Donghun Lee
The paper investigates how large language models activate a latent symbolic reasoning circuit—comprising abstraction, induction, and retrieval—when presented with in-context examples. By tracking this circuit across different shot counts and model families, the authors show that its components become detectable and functional long before the model reaches high accuracy. They further demonstrate that per-head causal contributions can increase eightfold from 1- to 10-shot, and that interventions such as cross-shot activation patching or function vector injection can dramatically improve accuracy, even at 0-shot, by leveraging the pre‑existing circuit in the model weights.
By Melissa Wessel
arXiv:2507. 11688v4 Announce Type: replace Abstract: Contemporary large models often exhibit behaviors suggesting the presence of low-level primitives that compose into modules with richer functionality, but these fundamental building blocks remain poorly understood.
By Travis Pence, Daisuke Yamada, Vikas Singh
arXiv:2511. 05963v4 Announce Type: replace Abstract: Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens.
By Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, John Langford
arXiv:2606. 06574v1 Announce Type: new Abstract: Large language models (LLMs) perform inference by following a fixed depth and order, non-recurrent execution of all layers.
By Ziyue Li, Yang Li, Tianyi Zhou
Subspace-Decomposed JEPAs (SD-JEPA) split the latent space of Joint-Embedding Predictive Architectures into two orthogonal subspaces: a low-dimensional progression subspace trained with a cosine-margin triplet loss and a high-dimensional content subspace regularised by SIGReg. The authors prove that the anti-collapse forces act on disjoint coordinates, allowing additive composition rather than competition. SD-JEPA outperforms the LeWM baseline on most control benchmarks and the strongest non-LeWM JEPA baseline on Push‑T, with a subspace-ablation confirming the split as essential. The 1‑D angular progression coordinate serves as a scene-aware compass, advancing with task progress, regressing on backtracking, and relocalising under perturbations to separate surprise from meaning.
By Lucas Thil, Jesse Read, Rim Kaddah, Guillaume Doquet
arXiv:2606. 27538v1 Announce Type: cross Abstract: We introduce the context-ready transformer, a new recurrent neural network architecture built from a D-layer transformer block that pre-contextualizes each token before it enters the block.
By Mahesh Godavarti
The paper introduces a neurosymbolic world model that separates observation reconstruction from reward prediction, enabling the model to adapt zero‑shot to new reward functions defined over a shared symbolic state space. This approach addresses the task‑dependency of traditional neural world models, which learn latent representations tied to specific training tasks. Experiments show that the neurosymbolic formulation generalises more strongly than purely neural methods.
By Isidoro Tamassia, Lennert De Smet, Giuseppe Marra
arXiv:2607. 13047v1 Announce Type: new Abstract: Parameter decomposition (PD) decomposes neural networks into interpretable computational components that faithfully reflect the original network's operations.
By Antoine Vigouroux, Lee Sharkey