arXiv Machine Learning October 23, 2026 By Ferdinand Kapl, Kaitlin Maile, Stefan Bauer, Johannes von Oswald

How Depth Growth and Looping Shape In-Context Learning in Language Models

Read the original on arXiv Machine Learning →

The paper investigates how depth growth and looping in language models influence in-context learning (ICL). It shows that models with these architectural features rely more on contextual information, acquire ICL earlier, and retain it better than fully dense counterparts. Attention analysis reveals that depth growth propagates early induction heads across layers, and ablations confirm that these models depend more on induction-ranked heads for few‑shot performance and token prediction.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Aug 11

Large language models reorganize representational geometry during in-context learning

arXiv:2605. 28854v2 Announce Type: replace-cross Abstract: Large language models (LLMs) exhibit remarkable flexibility in adapting to novel tasks from in-context examples without parameter updates, a capability known as in-context learning (ICL).

By Hua-Dong Xiong, Li Ji-An, Robert C. Wilson, Kwonjoon Lee, Xue-Xin Wei

llms

arXiv AI

Jul 22

Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning

arXiv:2607. 19345v1 Announce Type: cross Abstract: Large language models that generate step-by-step reasoning traces have achieved strong performance on complex tasks, and extending them to long-context settings has emerged as an important frontier.

By Lizhe Fang, Weizhou Shen, Tianyi Tang, Yisen Wang

llms reinforcement-learning benchmarks

arXiv AI

Jun 9

Emergence of Context Characteristics Sensitivity in Large Language Models

arXiv:2606. 09525v1 Announce Type: cross Abstract: During instruction fine-tuning (IFT), large language models (LLMs) learn to follow instructions by using the provided context to answer a query.

By Nadya Yuki Wangsajaya, Haeun Yu, Isabelle Augenstein

llms reinforcement-learning fine-tuning

arXiv AI

Aug 13

Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

arXiv:2608. 12218v1 Announce Type: cross Abstract: Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories.

By Arda Uzunoglu, Benjamin van Durme, Daniel Khashabi

llms fine-tuning

arXiv Computation and Language

Oct 23

In-Context Learning Beyond Copying: A Training-Time Analysis of Abstractive ICL

The paper investigates the relationship between in-context copying and the emergence of abstractive in-context learning (ICL) in large language models. By introducing Hapax, a training regime that removes loss contributions from tokens predictable by induction circuits, the authors show that abstractive ICL can still develop even when copying signals are largely suppressed. Attention analysis reveals that Hapax-trained models exhibit fewer and weaker induction heads, indicating a weaker developmental link between induction circuits and abstractive ICL than previously thought.

By Kerem Sahin, Sheridan Feucht, Adam Belfki, Jannik Brinkmann, Aaron Mueller, David Bau, Chris Wendler

llms

arXiv AI

Jul 3

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

arXiv:2607. 02509v1 Announce Type: new Abstract: Understanding and reasoning over long contexts has become a key requirement for deploying large language models (LLMs) in realistic applications.

By Yanjun Zhao, Ruizhong Qiu, Tianxin Wei, Yuanchen Bei, Zhining Liu, Lingjie Chen, Ismini Lourentzou, Hanghang Tong, Jingrui He

llms efficiency

One email a morning, machine-written

Related stories

Large language models reorganize representational geometry during in-context learning

Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning

Emergence of Context Characteristics Sensitivity in Large Language Models

Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

In-Context Learning Beyond Copying: A Training-Time Analysis of Abstractive ICL

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning