The study investigates how language models acquire in-context learning (ICL) by distinguishing between copying a nearby demonstration and integrating information across a prompt. Across 17 models, roughly one-third default to copied answers, a few are undecided, and the remainder favor rule-consistent answers that yield higher accuracy and are guided by the requested value rather than the nearest conflicting example. Analyses show that models prioritize task-relevant prompt tokens before accuracy surpasses chance, and that synthetic or fine-tuned training can shift models toward compositional responses or improve accuracy when prompts reassign label meanings.
arXiv:2607. 19345v1 Announce Type: cross Abstract: Large language models that generate step-by-step reasoning traces have achieved strong performance on complex tasks, and extending them to long-context settings has emerged as an important frontier.
arXiv:2610.14574v1 Announce Type: new Abstract: In-context learning (ICL) allows large language models to use demonstrations without updating their parameters. For interactive agents, demonstrations...
arXiv:2607. 03478v1 Announce Type: new Abstract: Post-training of frontier language models is conducted on curated task suites, and inevitably leaves a distribution shift between training and deployment environments.
arXiv:2609.34187v2 Announce Type: replace-cross Abstract: The strong version of the stochastic parrot argument claims that, although large language models (LLMs) may exceed rote regurgitation, they c...
arXiv:2606. 09525v1 Announce Type: cross Abstract: During instruction fine-tuning (IFT), large language models (LLMs) learn to follow instructions by using the provided context to answer a query.
arXiv:2602.02326v3 Announce Type: replace Abstract: If multilingual language model representations contain separable language-associated structure, then shifting between languages may require only si...
arXiv:2507. 04221v3 Announce Type: replace-cross Abstract: We introduce Context Tuning, a simple and effective method to significantly enhance few-shot adaptation of large language models (LLMs) without weight updates.
arXiv:2606. 32002v1 Announce Type: new Abstract: Language models are increasingly taught from synthetic question--answer (QA) supervision: a model generates questions about a document, answers them from the same text, and the resulting pairs are used to fine-tune, distill, or compress knowledge into another model.
The paper investigates how different prompt adaptation techniques—discrete prompt optimization, prompt tuning, and prefix tuning—affect the internal processing of dense and mixture-of-experts language models. It finds that the adaptation location determines how task information reaches the output: text prompts mainly influence response construction, input-level prompts modify content representations, and layer-wise prefixes exert direct control via attention. In MoE models, these differences also lead to distinct expert‑routing patterns, while generic representation‑level metrics fail to reliably distinguish effective adaptations from placebo effects.
The paper investigates how large language models learn new tasks in-context, comparing rule-based instruction following to example-based few-shot prompting across five diverse tasks. Results show that models generally learn more reliably from rule descriptions than from examples alone, and adding more examples does not consistently improve performance. Instruction tuning further enhances rule-based learning while preserving example-based capabilities, with rule advantages being strongest for algebraic tasks and weaker for tasks requiring distributional sensitivity or parametric knowledge.
arXiv:2608. 16627v1 Announce Type: cross Abstract: Natural language explanations (NLEs) are increasingly used as inputs, for example, as few-shot rationales that influence model behavior in in-context learning (ICL).
arXiv:2601. 06196v3 Announce Type: replace-cross Abstract: Large language models (LLMs) frequently generate factually incorrect or unsupported content, commonly referred to as hallucinations.