The paper investigates how depth growth and looping in language models influence in-context learning (ICL). It shows that models with these architectural features rely more on contextual information, acquire ICL earlier, and retain it better than fully dense counterparts. Attention analysis reveals that depth growth propagates early induction heads across layers, and ablations confirm that these models depend more on induction-ranked heads for few‑shot performance and token prediction.
arXiv:2605. 28854v2 Announce Type: replace-cross Abstract: Large language models (LLMs) exhibit remarkable flexibility in adapting to novel tasks from in-context examples without parameter updates, a capability known as in-context learning (ICL).
arXiv:2607. 19345v1 Announce Type: cross Abstract: Large language models that generate step-by-step reasoning traces have achieved strong performance on complex tasks, and extending them to long-context settings has emerged as an important frontier.
arXiv:2606. 09525v1 Announce Type: cross Abstract: During instruction fine-tuning (IFT), large language models (LLMs) learn to follow instructions by using the provided context to answer a query.
arXiv:2608. 12218v1 Announce Type: cross Abstract: Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories.
The paper investigates the relationship between in-context copying and the emergence of abstractive in-context learning (ICL) in large language models. By introducing Hapax, a training regime that removes loss contributions from tokens predictable by induction circuits, the authors show that abstractive ICL can still develop even when copying signals are largely suppressed. Attention analysis reveals that Hapax-trained models exhibit fewer and weaker induction heads, indicating a weaker developmental link between induction circuits and abstractive ICL than previously thought.
arXiv:2607. 02509v1 Announce Type: new Abstract: Understanding and reasoning over long contexts has become a key requirement for deploying large language models (LLMs) in realistic applications.
Understanding and reasoning over long contexts has become a key requirement for deploying large language models (LLMs) in realistic applications. Although recent LLMs support increasingly long context windows, they often fail to use relevant evidence that is already present in the input, revealing a gap between context access and effective context utilization.
arXiv:2602. 23197v2 Announce Type: replace-cross Abstract: Transformer-based large language models exhibit in-context learning, enabling adaptation to downstream tasks via few-shot prompting with demonstrations.
arXiv:2606. 03217v1 Announce Type: cross Abstract: Chain-of-thought (CoT) reasoning has become a widely used mechanism for eliciting multi-step reasoning in large language models by generating intermediate reasoning steps at inference time.
Chain-of-thought (CoT) reasoning has become a widely used mechanism for eliciting multi-step reasoning in large language models by generating intermediate reasoning steps at inference time. Yet the scaling behavior of generalization with CoT depth remains poorly understood.
arXiv:2507. 04221v3 Announce Type: replace-cross Abstract: We introduce Context Tuning, a simple and effective method to significantly enhance few-shot adaptation of large language models (LLMs) without weight updates.
arXiv:2505. 17315v2 Announce Type: replace Abstract: Recent language models exhibit strong reasoning capabilities, yet the influence of long-context capacity on reasoning remains underexplored.