The article examines whether large language models (LLMs) can effectively use representations they learn in-context to perform downstream tasks. Experiments with open‑weight LLMs on next‑token prediction and a new adaptive world modeling task reveal that these models struggle to deploy novel semantics defined in-context, even when such semantics are encoded in their latent representations. The study also shows that state‑of‑the‑art closed‑source reasoning models similarly fail to reliably leverage in‑context patterns, highlighting a gap in flexible representation usage.
By Michael A. Lepori, Tal Linzen, Ann Yuan, Katja Filippova
The paper investigates how large language models learn new tasks in-context, comparing rule-based instruction following to example-based few-shot prompting across five diverse tasks. Results show that models generally learn more reliably from rule descriptions than from examples alone, and adding more examples does not consistently improve performance. Instruction tuning further enhances rule-based learning while preserving example-based capabilities, with rule advantages being strongest for algebraic tasks and weaker for tasks requiring distributional sensitivity or parametric knowledge.
By Xiang Fu, Seungmin Cho, Yukyung Lee, Najoung Kim
Task-Level Natural Language Priors as Learning Signals for Low-Resource LLM Training proposes Prior-Guided Tuning (PGT), a training approach that treats natural-language priors as auxiliary learning signals rather than just input context. The method introduces Contrastive Prior Steering (CPS), which adds positive and negative prior-conditioned auxiliary losses while preserving the original supervised objective. Experiments on AmbiMath, Jigsaw, and MNLI/HANS demonstrate that CPS consistently outperforms plain and prompt fine-tuning, achieving high accuracy and significant gains with limited training data.
By Jian Gao, Xiao Zhang, Xun Zhu, Miao Li, Ji Wu
arXiv:2609.36788v1 Announce Type: new
Abstract: Incorporating rich task-relevant context, such as domain knowledge and external observations, is a key capability yet remains challenging for Bayesian...
By Zhongwei Yu, Sourabh Roy, Bin Cao, Xue Yan, Anjie Liu, Jun Wang
arXiv:2608.23475v1 Announce Type: new
Abstract: As large language models are increasingly used in data-scarce and evolving task scenarios, few-shot in-context learning (ICL) has become a key paradigm...
By Jinghan Tan, Yuanzheng Wang, Lu Chen, Zijun Chen, Yuqian Wang, Maosong Sun
The paper investigates when in‑context learning (ICL) can succeed even if the instruction or target task is not covered by a model’s pretraining data. It introduces a theoretical framework that distinguishes two types of distribution shift—unsupported instructions and unsupported target tasks—and shows how demonstrations can mitigate these shifts. The authors further extend the analysis to long‑horizon tasks that can be decomposed into pretraining‑supported subtasks, providing error bounds that explain how ICL can handle tasks beyond the model’s original support.
By Xuhan Tong, Yuchen Zeng, Jiawei Zhang
arXiv:2509.24988v2 Announce Type: replace-cross
Abstract: Generating accurate and calibrated confidence estimates is critical for deploying LLMs in high-stakes or user-facing applications, and remain...
By Hanqi Xiao, Vaidehi Patil, Hyunji Lee, Elias Stengel-Eskin, Mohit Bansal
arXiv:2604. 01161v2 Announce Type: replace Abstract: Large language models (LLMs) exhibiting test-time scaling behavior, such as extended reasoning traces and self-verification, have demonstrated remarkable performance on complex, long-term reasoning tasks.
By Gleb Rodionov, Roman Garipov, George Yakushev
arXiv:2601. 20379v2 Announce Type: replace Abstract: Large language models (LLMs) struggle with complex, long-horizon reasoning due to instability caused by their frozen policy assumption.
By Zhengbo Jiao, Hongyu Xian, Qinglong Wang, Yunpu Ma, Zhebo Wang, Zifan Zhang, Dezhang Kong, Meng Han
arXiv:2609.00759v1 Announce Type: new
Abstract: Large language models (LLMs) increasingly handle in-context learning (ICL) tasks where a long, novel context defines the rules, knowledge, and output s...
By Jinhu Qi, Minda Hu, Wentao Zhang, Weiqiang Jin, Yanyu Chen, Junli Wang, Irwin King
The paper investigates how biases present in in-context examples used by Large Language Models (LLMs) for synthetic tabular data generation can propagate to the generated data, causing global statistical distortions even when the biases are mild. It also explores an adversarial scenario where a malicious actor can inject bias through a subset of examples, thereby compromising the fairness of downstream classifiers for protected subgroups. The authors evaluate prompt-side mitigation strategies, such as preprocessing examples and instruction-based defenses, noting that while these can reduce disparity, LLMs remain highly sensitive to adversarial prompts, revealing a significant vulnerability in data generation pipelines for sensitive domains.
By Pol G. Recasens, Alberto Gutierrez, Jordi Torres, Josep. Ll Berral, Javier Carnerero-Cano, Anisa Halimi, Kieran Fraser
arXiv:2606. 30704v1 Announce Type: cross Abstract: Large language models (LLMs) excel across a wide range of tasks, yet their instance-specific solutions often lack the structural consistency needed for reliable deployment.
By Gan Luo, Zihan Qin, Bin Dong, Wotao Yin