← Back to all news
arXiv AI By Ha Van Dau, Thanh Tung Khuat, Nguyen Thanh Dung

Beyond Depth Truncation: Controlled Evaluation of Depth Utilization in Recursive Language Models

Read the original on arXiv AI →

The paper critiques the common practice of evaluating depth usage in depth‑recurrent language models by truncating depth during inference and measuring performance decline. It argues that this method conflates three distinct effects—fewer block applications, reduced computation, and an out‑of‑distribution readout—yet is usually interpreted as measuring only the second. To address this, the authors introduce the Depth Control Protocol (DCP), a suite of positive and negative controls that isolate each factor, along with a training intervention to confirm causality, specifically tailored for depth‑wise weight‑sharing architectures.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 15

Bypass Observation: A Conceptual Design of a Non-Intrusive Layer-Wise Semantic Extraction Architecture

The paper proposes Bypass Observation, a non‑intrusive layer‑wise readout architecture that attaches read‑only observation heads to selected Transformer layers without feeding their outputs back into the backbone. Three variants are explored: a shared language‑model head across layers, layer‑specific heads, and a layer‑ or step‑adaptive head. The authors provide a closed‑form overhead estimate (≈ V/(12d)) and discuss ways to reduce cost, while distinguishing bypass chain‑of‑thought from conventional chain‑of‑thought and outlining potential applications to looped and recurrent‑depth Transformers.

By Haibin Tong, Jiang Yu
llmsreinforcement-learning
More like this →
arXiv AI
Jul 16

DeepLoop: Depth Scaling for Looped Transformers

arXiv:2607. 13491v1 Announce Type: cross Abstract: Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters.

By Shuzhen Li, Yifan Zhang, Jiacheng Guo, Quanquan Gu, Mengdi Wang
llmssafety
More like this →
arXiv Machine Learning
Aug 20

Allocating Recurrent Compute in Looped Language Models

The paper proposes MixerLoop, a looped language model that repeats only the Gated DeltaNet mixer while applying the dense feed‑forward network (FFN) once, instead of repeating the entire layer stack. Using Iterative Transport Rank (ITR) to measure cross‑position influence, the authors show that MixerLoop retains most of the performance gains of full recurrence while reducing FLOPs by 45.9%. Experiments on 15M and 110M parameter models demonstrate that MixerLoop outperforms full‑block recurrence on aggregate CORE metrics.

By Ruhai Lin, Yiyang Guo, Rui-Jie Zhu, Hao Ye, Jason K. Eshraghian
llmsrobotics
More like this →