arXiv Machine Learning July 2, 2026 By Peilin Liu, Ding-Xuan Zhou

Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

Read the original on arXiv Machine Learning →

arXiv:2607. 00479v1 Announce Type: new Abstract: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Sep 25

Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning

Transformers can learn broad families of tasks during pretraining and adapt to unseen tasks from a short prompt, but a rigorous understanding of this capability is limited. This paper studies how shared cross‑task structure influences the sample complexity of in‑context learning (ICL) by characterizing task‑space complexity through covering numbers, yielding a set of anchor functions that localize unseen tasks and predict responses. The authors construct a Transformer with Softmax attention to approximate this procedure and derive an error bound that separates the effects of pretraining tasks and prompt length, showing that once enough tasks are available the dependence on prompt length becomes dimension‑free.

By Zhongjie Shi, Rongjie Lai, Alexander Cloninger, Wenjing Liao

llms

arXiv Machine Learning

Jun 30

Generalization Analysis of Transformers in Distribution Regression

arXiv:2606. 29256v1 Announce Type: cross Abstract: In recent years, models based on the Transformer architecture have seen widespread applications and have become one of the core tools in the field of deep learning.

By Peilin Liu, Ding-Xuan Zhou

llms nlp fine-tuning

arXiv Machine Learning

Aug 11

Training-Free Universal Approximation by Prompting Random Transformers

arXiv:2608. 09558v1 Announce Type: new Abstract: How expressive is prompting a transformer?

By Alexander Hsu, Rongjie Lai

llms

arXiv Machine Learning

Jul 7

Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

arXiv:2607. 03660v1 Announce Type: cross Abstract: Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt.

By Mary Letey, Yue M. Lu, Cengiz Pehlevan, Jacob Zavatone-Veth

llms

arXiv Machine Learning

Jun 17

Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective

arXiv:2512. 11784v2 Announce Type: replace Abstract: Softmax attention is a central component of transformer architectures, yet its nonlinear structure poses significant challenges for theoretical analysis.

By Etienne Boursier, Claire Boyer

llms

arXiv AI

Jun 16

Adaptive inference and function vectors in deep transformers

arXiv:2606. 16694v1 Announce Type: cross Abstract: Transformers are widely used as a general-purpose substrate for learning complex correlations between a large collection of coupled variables, but their internal mechanisms have remained mysterious.

By Ravin Raj, Gautam Reddy

llms

One email a morning, machine-written

Related stories

Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning

Generalization Analysis of Transformers in Distribution Regression

Training-Free Universal Approximation by Prompting Random Transformers

Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective

Adaptive inference and function vectors in deep transformers