arXiv Machine Learning July 7, 2026 By Chenrui Liu, Chuanlong Xie, Falong Tan, Yicheng Zeng, Lixing Zhu

A Unified Framework for In-Context Learning with Causal and Masked Language Models

Read the original on arXiv Machine Learning →

arXiv:2607. 04081v1 Announce Type: new Abstract: In-context learning (ICL) has emerged as a central capability of pretrained language models, yet its theoretical analysis has focused primarily on causal language models trained by left-to-right autoregressive prediction, such as GPT-style models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Jun 25

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off

arXiv:2510. 01163v2 Announce Type: replace Abstract: The factors driving the performance of in-context learning (ICL) in large language models (LLMs) remain poorly understood despite ICL's surprising effectiveness, enabling models to adapt to new tasks from only a handful of examples.

By Wa\"iss Azizian, Ali Hasan

llms

arXiv Machine Learning

Sep 22

ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining

arXiv:2505.19893v2 Announce Type: replace Abstract: Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency. We introduce E...

By Melis Ilayda Bal, Volkan Cevher, Michael Muehlebach

llms efficiency

arXiv AI

Jul 2

CausalMix: Data Mixture as Causal Inference for Language Model Training

arXiv:2607. 01104v1 Announce Type: cross Abstract: In Large Language Model (LLM) training, data mixing plays a pivotal role in determining model performance.

By Zinan Tang, Yukun Zhang, Shaomian Zheng, Zhuoshi Pan, Qizhi Pei, Dingnan Jin, Jun Zhou, Yujun Wang, Biqing Huang

llms

arXiv AI

Jun 8

Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation

arXiv:2606. 06712v1 Announce Type: cross Abstract: We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs).

By Xingyu Su, Jacob Helwig, Shubham Parashar, Atharv Chagi, Lakshmi Jotsna, Degui Zhi, James Caverlee, Dileep Kalathil, Shuiwang Ji

llms diffusion efficiency

arXiv Machine Learning

Jun 16

In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning

arXiv:2510. 10981v3 Announce Type: replace-cross Abstract: This paper develops a finite-sample statistical theory for in-context learning (ICL), analyzed within a meta-learning framework that accommodates mixtures of diverse task types.

By Tomoya Wakayama, Taiji Suzuki

llms

arXiv Machine Learning

Jul 21

Bigger Is Safer: Provable Robustness in In-Context Learning Scales with Capacity

arXiv:2602. 17743v2 Announce Type: replace Abstract: In-context learning (ICL) allows large language models to adapt to new tasks from a few examples without updating their parameters.

By Di Zhang, Ningxu Zhang, Zimeng Liu

llms safety

One email a morning, machine-written

Related stories

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off

ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining

CausalMix: Data Mixture as Causal Inference for Language Model Training

Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation

In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning

Bigger Is Safer: Provable Robustness in In-Context Learning Scales with Capacity