arXiv Machine Learning September 25, 2026 By Zhongjie Shi, Rongjie Lai, Alexander Cloninger, Wenjing Liao

Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning

Read the original on arXiv Machine Learning →

Transformers can learn broad families of tasks during pretraining and adapt to unseen tasks from a short prompt, but a rigorous understanding of this capability is limited. This paper studies how shared cross‑task structure influences the sample complexity of in‑context learning (ICL) by characterizing task‑space complexity through covering numbers, yielding a set of anchor functions that localize unseen tasks and predict responses. The authors construct a Transformer with Softmax attention to approximate this procedure and derive an error bound that separates the effects of pretraining tasks and prompt length, showing that once enough tasks are available the dependence on prompt length becomes dimension‑free.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Jul 2

Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

arXiv:2607. 00479v1 Announce Type: new Abstract: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning.

By Peilin Liu, Ding-Xuan Zhou

llms

arXiv Machine Learning

Jun 8

The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspaces

arXiv:2606. 06814v1 Announce Type: cross Abstract: The transformer's emergent ability to perform in-context learning (ICL) has sparked a wide range of studies designed to understand its underlying mechanisms.

By Soo Min Kwon, Alec S. Xu, Can Yaras, Dogyoon Song, Laura Balzano, Qing Qu

llms

arXiv Machine Learning

Jun 25

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off

arXiv:2510. 01163v2 Announce Type: replace Abstract: The factors driving the performance of in-context learning (ICL) in large language models (LLMs) remain poorly understood despite ICL's surprising effectiveness, enabling models to adapt to new tasks from only a handful of examples.

By Wa\"iss Azizian, Ali Hasan

llms

arXiv Machine Learning

Jun 16

In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning

arXiv:2510. 10981v3 Announce Type: replace-cross Abstract: This paper develops a finite-sample statistical theory for in-context learning (ICL), analyzed within a meta-learning framework that accommodates mixtures of diverse task types.

By Tomoya Wakayama, Taiji Suzuki

llms

arXiv Machine Learning

Jul 7

Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

arXiv:2607. 03660v1 Announce Type: cross Abstract: Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt.

By Mary Letey, Yue M. Lu, Cengiz Pehlevan, Jacob Zavatone-Veth

llms

arXiv Machine Learning

Jun 2

Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models

arXiv:2602. 23197v2 Announce Type: replace-cross Abstract: Transformer-based large language models exhibit in-context learning, enabling adaptation to downstream tasks via few-shot prompting with demonstrations.

By Chungpa Lee, Jy-yong Sohn, Kangwook Lee

llms fine-tuning

One email a morning, machine-written

Related stories

Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspaces

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off

In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning

Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models