arXiv Computation and Language
Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Oct 2
Closing the Loop: Practical Training Recipes for Looped Language Models
The paper presents practical training recipes for looped language models, showing that a compute‑efficient pipeline can reduce the training budget from 7.7 T tokens to 310 B tokens while maintaining strong reasoning performance. In controlled experiments, a 1.4 B LoopLM outperforms a parameter‑matched dense model on 12 benchmarks, achieving significant gains on GSM8K, MATH, and DROP, and approaches a 3.9 B dense model at matched inference compute. The authors also provide a minimal recipe to convert pretrained dense models into looped ones, demonstrating improvements on Qwen3‑1.7B‑Base across multiple data regimes.
arXiv AI
Jul 14
Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
arXiv:2508. 10123v3 Announce Type: replace-cross Abstract: Advanced reasoning in LLMs on challenging domains like mathematical reasoning can be tackled using verifiable rewards based reinforced fine-tuning (ReFT).
arXiv Computation and Language
Sep 30
Pretraining Latent Information Feedback Transformers with Teacher Supervision
arXiv:2609.38149v1 Announce Type: new Abstract: Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for informa...
arXiv AI
Sep 2
Instella-MoE Technical Report
Instella‑MoE is a fully open Mixture‑of‑Experts language model with 16 billion total parameters and 2.8 billion active parameters per token, trained from scratch on AMD Instinct GPUs. It incorporates a sparsely activated MoE design with Gated Multi‑head Latent Attention and FarSkip‑Collective connectivity, and follows a multi‑stage pipeline that includes pre‑training, long‑context extension, supervised fine‑tuning, direct preference optimization, and reinforcement learning with Multi‑Teacher On‑Policy Distillation. The model achieves an average score of 76.7 on pre‑training benchmarks and 73.2 on instruction‑following, reasoning, math, coding, and chat benchmarks, outperforming comparable fully open and open‑weight models, and its full training pipeline, weights, and code are released for reproducibility.
arXiv Machine Learning
Sep 30
What Makes Recurrence Effective in Looped Language Models?
arXiv:2609.36636v1 Announce Type: new Abstract: Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding...
arXiv AI
Jun 19
Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts
arXiv:2402. 14035v4 Announce Type: replace-cross Abstract: Knowledge distillation from foundation models to compact domain models is challenging due to substantial gaps in capacity, architecture, and modality.