The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The article examines whether large language models (LLMs) can effectively use representations they learn in-context to perform downstream tasks. Experiments with open‑weight LLMs on next‑token prediction and a new adaptive world modeling task reveal that these models struggle to deploy novel semantics defined in-context, even when such semantics are encoded in their latent representations. The study also shows that state‑of‑the‑art closed‑source reasoning models similarly fail to reliably leverage in‑context patterns, highlighting a gap in flexible representation usage.
The paper investigates how large language models learn new tasks in-context, comparing rule-based instruction following to example-based few-shot prompting across five diverse tasks. Results show that models generally learn more reliably from rule descriptions than from examples alone, and adding more examples does not consistently improve performance. Instruction tuning further enhances rule-based learning while preserving example-based capabilities, with rule advantages being strongest for algebraic tasks and weaker for tasks requiring distributional sensitivity or parametric knowledge.
Task-Level Natural Language Priors as Learning Signals for Low-Resource LLM Training proposes Prior-Guided Tuning (PGT), a training approach that treats natural-language priors as auxiliary learning signals rather than just input context. The method introduces Contrastive Prior Steering (CPS), which adds positive and negative prior-conditioned auxiliary losses while preserving the original supervised objective. Experiments on AmbiMath, Jigsaw, and MNLI/HANS demonstrate that CPS consistently outperforms plain and prompt fine-tuning, achieving high accuracy and significant gains with limited training data.
arXiv:2609.36788v1 Announce Type: new Abstract: Incorporating rich task-relevant context, such as domain knowledge and external observations, is a key capability yet remains challenging for Bayesian...
arXiv:2608.23475v1 Announce Type: new Abstract: As large language models are increasingly used in data-scarce and evolving task scenarios, few-shot in-context learning (ICL) has become a key paradigm...
The paper investigates when in‑context learning (ICL) can succeed even if the instruction or target task is not covered by a model’s pretraining data. It introduces a theoretical framework that distinguishes two types of distribution shift—unsupported instructions and unsupported target tasks—and shows how demonstrations can mitigate these shifts. The authors further extend the analysis to long‑horizon tasks that can be decomposed into pretraining‑supported subtasks, providing error bounds that explain how ICL can handle tasks beyond the model’s original support.