arXiv Machine Learning June 25, 2026 By Wa\"iss Azizian, Ali Hasan

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off

Read the original on arXiv Machine Learning →

arXiv:2510. 01163v2 Announce Type: replace Abstract: The factors driving the performance of in-context learning (ICL) in large language models (LLMs) remain poorly understood despite ICL's surprising effectiveness, enabling models to adapt to new tasks from only a handful of examples.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Jul 7

A Unified Framework for In-Context Learning with Causal and Masked Language Models

arXiv:2607. 04081v1 Announce Type: new Abstract: In-context learning (ICL) has emerged as a central capability of pretrained language models, yet its theoretical analysis has focused primarily on causal language models trained by left-to-right autoregressive prediction, such as GPT-style models.

By Chenrui Liu, Chuanlong Xie, Falong Tan, Yicheng Zeng, Lixing Zhu

llms

arXiv Machine Learning

Jun 16

In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning

arXiv:2510. 10981v3 Announce Type: replace-cross Abstract: This paper develops a finite-sample statistical theory for in-context learning (ICL), analyzed within a meta-learning framework that accommodates mixtures of diverse task types.

By Tomoya Wakayama, Taiji Suzuki

llms

arXiv Machine Learning

Jul 1

Probing Memorization of Tabular In-Context Learning

arXiv:2606. 31208v1 Announce Type: new Abstract: Large tabular models (LTMs), i.

By Francesco Capano, Jonas B\"ohler

llms fine-tuning benchmarks

arXiv Machine Learning

Aug 11

Large language models reorganize representational geometry during in-context learning

arXiv:2605. 28854v2 Announce Type: replace-cross Abstract: Large language models (LLMs) exhibit remarkable flexibility in adapting to novel tasks from in-context examples without parameter updates, a capability known as in-context learning (ICL).

By Hua-Dong Xiong, Li Ji-An, Robert C. Wilson, Kwonjoon Lee, Xue-Xin Wei

llms

arXiv AI

6d ago

Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

arXiv:2608. 12218v1 Announce Type: cross Abstract: Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories.

By Arda Uzunoglu, Benjamin van Durme, Daniel Khashabi

llms fine-tuning

arXiv AI

Jul 28

Extracting Algorithms in Pre-trained LLMs: A Case on Hidden Markov Models

arXiv:2607. 22646v1 Announce Type: new Abstract: Large language models (LLMs) display a striking ability to predict next observations from Hidden Markov Models (HMMs) via in-context learning (ICL), but the algorithm underlying this capability remains undetermined: prior work has proposed several candidates without consensus, and none has been grounded in the model's internal activations.

By Yijia Dai, Zhaolin Gao, Yahya Sattar, Jennifer J. Sun, Sarah Dean

llms

Related stories

A Unified Framework for In-Context Learning with Causal and Masked Language Models

In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning

Probing Memorization of Tabular In-Context Learning

Large language models reorganize representational geometry during in-context learning

Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

Extracting Algorithms in Pre-trained LLMs: A Case on Hidden Markov Models