arXiv Machine Learning July 7, 2026 By Mary Letey, Yue M. Lu, Cengiz Pehlevan, Jacob Zavatone-Veth

Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

Read the original on arXiv Machine Learning →

arXiv:2607. 03660v1 Announce Type: cross Abstract: Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Jul 2

Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

arXiv:2607. 00479v1 Announce Type: new Abstract: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning.

By Peilin Liu, Ding-Xuan Zhou

llms

arXiv Machine Learning

Sep 25

Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning

Transformers can learn broad families of tasks during pretraining and adapt to unseen tasks from a short prompt, but a rigorous understanding of this capability is limited. This paper studies how shared cross‑task structure influences the sample complexity of in‑context learning (ICL) by characterizing task‑space complexity through covering numbers, yielding a set of anchor functions that localize unseen tasks and predict responses. The authors construct a Transformer with Softmax attention to approximate this procedure and derive an error bound that separates the effects of pretraining tasks and prompt length, showing that once enough tasks are available the dependence on prompt length becomes dimension‑free.

By Zhongjie Shi, Rongjie Lai, Alexander Cloninger, Wenjing Liao

llms

arXiv Machine Learning

Jul 8

How Can Mamba Learn In Context with Outliers and Generalize Provably?

arXiv:2510. 00399v2 Announce Type: replace Abstract: The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of language tasks.

By Hongkang Li, Songtao Lu, Xiaodong Cui, Pin-Yu Chen, Meng Wang

llms fine-tuning

arXiv Machine Learning

4d ago

Theory on Attention Dynamics for Out-of-Distribution In-Context Learning

arXiv:2609.36448v1 Announce Type: new Abstract: Transformers have demonstrated remarkable in-context learning (ICL) capabilities, enabling them to perform new tasks without additional fine-tuning. Ho...

By Junze Deng, Daouda Sow, Sen Lin, Yingbin Liang

llms fine-tuning

arXiv Machine Learning

Jul 7

Induction Heads Interpolate N-Grams

arXiv:2607. 02800v1 Announce Type: new Abstract: Induction heads are attention circuits believed to underlie in-context learning in transformers, yet a precise characterization of the estimators they implement remains elusive.

By Francesco D'Angelo, Oguz Kaan Yuksel, Swathi Shree Narashiman, Nicolas Flammarion

llms safety

arXiv Machine Learning

Jul 30

Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training

arXiv:2505. 15548v2 Announce Type: replace Abstract: Autoregressive transformer language models frequently exhibit training instability when trained on long sequences, particularly under low-precision arithmetic.

By Suvadeep Hajra

llms

One email a morning, machine-written

Related stories

Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning

How Can Mamba Learn In Context with Outliers and Generalize Provably?

Theory on Attention Dynamics for Out-of-Distribution In-Context Learning

Induction Heads Interpolate N-Grams

Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training