← Back to all news
arXiv Machine Learning

Recursive Language Models Generalize Out of Domain

The paper investigates how restricting a language model’s view to isolated subtask contexts—via recursive language models—affects learning compared to standard chain‑of‑thought (CoT) approaches. While CoT can simulate recursive reasoning and offers similar in‑distribution performance, it tends to exploit contextual shortcuts that fail out of distribution; recursive models avoid this by isolating context, thereby preventing such shortcuts. The study shows that merely covering the correct rule is insufficient for robust reasoning beyond the training distribution.

Related stories

arXiv Computation and Language
Sep 4

LLMs Learn Better In-Context from Rules than from Examples

The paper investigates how large language models learn new tasks in-context, comparing rule-based instruction following to example-based few-shot prompting across five diverse tasks. Results show that models generally learn more reliably from rule descriptions than from examples alone, and adding more examples does not consistently improve performance. Instruction tuning further enhances rule-based learning while preserving example-based capabilities, with rule advantages being strongest for algebraic tasks and weaker for tasks requiring distributional sensitivity or parametric knowledge.

By Xiang Fu, Seungmin Cho, Yukyung Lee, Najoung Kim
llmsfine-tuning
More like this →
arXiv Machine Learning
Jun 29

Learning to Reason with Curriculum II: Compositional Generalization

arXiv:2606. 27721v1 Announce Type: new Abstract: Compositional generalization, the ability to solve complex problems by combining solutions to simpler sub-problems, is a fundamental capability of both natural and artificial intelligence, and a key mechanism underlying chain-of-thought reasoning.

By Nived Rajaraman, Audrey Huang, Miroslav Dudik, Robert Schapire, Dylan Foster, Akshay Krishnamurthy
reinforcement-learningfine-tuning
More like this →
arXiv AI
Jun 26

Where Do CoT Training Gains Land in LLM based Agents?

arXiv:2606. 26935v1 Announce Type: new Abstract: Chain-of-thought (CoT) reasoning is widely used in language-model agents, but prior work has shown that verbalized CoT is not always faithful and may instead reflect post-hoc reasoning, which means the model already knows the answer before reasoning.

By Jingyu Liu, Zhiwen Wang, Yuxin Jing, Huanyu Zhou, Yong Liu
llmsagents
More like this →
arXiv Machine Learning
Oct 23

How Depth Growth and Looping Shape In-Context Learning in Language Models

The paper investigates how depth growth and looping in language models influence in-context learning (ICL). It shows that models with these architectural features rely more on contextual information, acquire ICL earlier, and retain it better than fully dense counterparts. Attention analysis reveals that depth growth propagates early induction heads across layers, and ablations confirm that these models depend more on induction-ranked heads for few‑shot performance and token prediction.

By Ferdinand Kapl, Kaitlin Maile, Stefan Bauer, Johannes von Oswald
llmssafety
More like this →
arXiv AI
Sep 3

Language Models Can Control Their Own Attention

The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.

By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
llmsefficiency
More like this →