arXiv:2610.15218v1 Announce Type: new Abstract: A recurrent-depth language model reuses one block of weights several times in a row. The number of repetitions, which we call loops, is chosen when the...
The paper introduces Elastic Latent Recurrence (ELR), a training recipe that enables looped language models to match the accuracy of non-recurrent models at a single loop while allowing additional loops to improve accuracy with more compute. ELR achieves this through gated state updates with learned step sizes, mixed loop counts during training, and conditioning on loop index and budget. The method also reduces training and inference overhead in Mixture-of-Experts models and produces a single checkpoint, Koi‑4B, that performs competitively with state‑of‑the‑art models on multiple benchmarks.
arXiv:2607. 10203v2 Announce Type: replace-cross Abstract: Adaptive-compute world models -- early-exit or mixture-of-depths predictors that spend variable depth per step -- assume depth buys better predictions and can be routed adaptively.
LayerRoPE introduces a depth‑wise magnitude and angular superposition mechanism that replaces per‑layer normalization gains with a single shared vector and depth‑conditioned scalars, reducing parameters and computational cost. Across 16 pre‑trained large language models and a 100B+ token model ladder, LayerRoPE outperforms traditional Pre‑, Post‑, and Peri‑Norm schemes, achieving comparable or better performance with significantly less compute and improved convergence up to 512 layers. The method demonstrates that widening the residual stream, rather than shrinking it, enhances depth stability and learning‑rate sensitivity, and it transfers effectively to latent and vision transformer architectures.
The paper investigates whether adding a loop—re‑applying a block of layers—to a trained language model is worth the extra inference cost. Experiments on Qwen3 and Qwen3.5 models from 0.8B to 9B parameters show that only the 0.8B looped model outperforms both a non‑looped model with the same parameters and a model with the same FLOPs. For larger models (4B and 9B), the looped version does not beat the baseline trained without loops, indicating that loops only pay off for small models with a fixed parameter budget.
arXiv:2607. 10203v1 Announce Type: cross Abstract: Adaptive-compute world models -- early-exit or mixture-of-depths predictors that spend variable depth per step -- assume depth buys better predictions and can be routed adaptively.
The paper critiques the common practice of evaluating depth usage in depth‑recurrent language models by truncating depth during inference and measuring performance decline. It argues that this method conflates three distinct effects—fewer block applications, reduced computation, and an out‑of‑distribution readout—yet is usually interpreted as measuring only the second. To address this, the authors introduce the Depth Control Protocol (DCP), a suite of positive and negative controls that isolate each factor, along with a training intervention to confirm causality, specifically tailored for depth‑wise weight‑sharing architectures.
The paper investigates whether deeper predictor blocks in adaptive‑compute world models truly improve rollout accuracy and whether a router can select the appropriate depth per step. Experiments on nine DeepMind Control tasks show that deeper predictors do not always yield better accuracy, and that a greedy oracle selecting depth per step can reduce error by up to 37% compared to fixed‑depth mixtures, yet no trained router achieves this advantage. The study also finds that supervising early exits only at the first rollout step can mitigate depth‑inversion issues observed in some tasks.
RecurTrace introduces adaptive latent reasoning for language models by allowing each looped layer to attend to its own past states and by using a halting head to decide when to stop iterating. This approach overcomes two limitations of prior latent recurrence methods: limited access to earlier computations and a fixed loop count that mismatches input difficulty. In experiments on MathQA, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, outperforming fixed‑depth baselines and other adaptive methods, and it also improves generation accuracy across a range of model sizes.
arXiv:2610.19768v1 Announce Type: new Abstract: Looped transformers compute by applying one block repeatedly, so they can run for more loops than they were trained with. Recurrent reasoners are known...
The paper introduces Layerwise Repacking, a technique that eliminates an extra data redistribution step when training hybrid‑attention models with variable‑length sequences. By keeping full‑attention layers in wide sequence‑parallel groups (SP16/32) and repacking samples into node‑local sparse children (SP1/2/4/8) within the same all‑to‑all communication, the method preserves hidden‑state continuity and avoids costly inter‑node transfers. Experiments on a 1F3S hybrid‑attention model show that repacking alone cuts end‑to‑end training time by ~6 % and boosts FLOP utilization, with an improved solver further reducing time by ~7.6 % and increasing utilization by ~4.8 %.
arXiv:2610.14998v1 Announce Type: cross Abstract: Tabular foundation models adapt to new datasets through in-context learning (ICL), and, like most transformers, they tie computational depth to the n...