The paper introduces Prompt-agnostic Language Models, focusing on train‑time techniques to embed prompt robustness during instruction tuning. It shows that mixing paraphrases in training causes significant gradient interference—57% of per‑paraphrasing gradients conflict—leading to the development of two gradient‑merging methods, Prompt‑TIES and Prompt‑DARE, which resolve this interference. Experiments across four models demonstrate that these methods substantially reduce per‑example variance and improve local robustness with minimal impact on overall accuracy.
By Frederic Sadrieh
The paper investigates how small lexical changes in prompts can cause large performance swings in large language models. Using a dataset of 132,000 prompt variants, the authors uncover a scaling law linking higher average task performance to lower variance and greater robustness. They identify domain-specific terminology and explicit action directives as key linguistic factors that stabilize prompts, and propose an automated Prompt-Refining Agent that reduces performance variance by 40.7% in code generation while maintaining or improving mean performance.
By Qipeng Xie, Zi Liang, Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, Haiqin Yang, Kaishun Wu
As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy.
Large language models can serve as capable long-horizon agents, but their out-of-distribution (OOD) generalization remains weak. We identify a key source of this failure as task insensitivity: when faced with similar but distinct tasks, models might apply patterns learned during training and fail to solve the task at hand.
arXiv:2609.15209v1 Announce Type: cross
Abstract: Automatic prompt optimization (APO) improves language-model programs by revising prompts from task feedback, yet it typically holds its training data...
By Tianyu Yuan, Zhuzhong Qian
arXiv:2606. 26918v1 Announce Type: new Abstract: Large language models can serve as capable long-horizon agents, but their out-of-distribution (OOD) generalization remains weak.
By Jingyu Liu, Xiaopeng Wu, Kehan Chen, Chuan Yu, Yong Liu
arXiv:2609.23716v1 Announce Type: cross
Abstract: Textual-gradient methods automate prompt optimization through natural-language feedback, but their iterative updates can be unstable. We identify two...
By Yifan Xu, Yixuan Li, Xinzhuo Li, Yixin Gu, Yifan Shen, Lijun Yu, Haohan Wang
arXiv:2507. 09839v2 Announce Type: replace Abstract: An increasing number of NLP applications interact with large language models (LLMs) through black-box APIs, making prompt engineering critical for controlling model behavior.
By MohammadReza Davari, Utkarsh Garg, Weixin Cai, Eugene Belilovsky
arXiv:2610.01428v1 Announce Type: cross
Abstract: Generalization in large language models (LLMs) is the ability to produce consistent and semantically stable outputs when the same input is expressed...
By Nagham Omar, Mahmoud Jabarin, Maya Rozenshtein, Rom Himelstein, Avi Mendelson, Amit LeVi
The paper introduces a fine-grained method called interactions to analyze prompt sensitivity in large language models (LLMs). By decomposing output scores into nonlinear interactions, the authors show that subtle prompt changes can destabilize these interactions even when overall outputs stay unchanged. They propose an Interaction-based Prompt Sensitivity (IPS) metric and use it to evaluate 50 open-source LLMs, finding that supervised fine‑tuning, larger model scales, dense architectures, and few‑shot learning all reduce prompt sensitivity, primarily by stabilizing low‑order interactions.
By Ruiyang Qin, Qingzhuo Wang, Tian Wang, Zhihua Wei, Wen Shen
The paper introduces soft‑prompt tuning as a lightweight, automated method for evaluating large language models (LLMs). By optimizing only ten soft‑prompt vectors—about 0.0006% of a 7B model’s parameters—within 150 steps, the approach adapts models to specific benchmark formats, improving format compliance while preserving knowledge accuracy. Experiments across seven models and eleven datasets show that this method matches strong evaluation‑harness baselines, enhances evaluation signals on challenging datasets, and yields cleaner outputs that align better with exact‑match scoring and independent LLM judges.
By Selen Erkan, Bastian Boll, Kristian Kersting, Bj\"orn Deiseroth, Letitia Parcalabescu
The study investigates how language models acquire in-context learning (ICL) by distinguishing between copying a nearby demonstration and integrating information across a prompt. Across 17 models, roughly one-third default to copied answers, a few are undecided, and the remainder favor rule-consistent answers that yield higher accuracy and are guided by the requested value rather than the nearest conflicting example. Analyses show that models prioritize task-relevant prompt tokens before accuracy surpasses chance, and that synthetic or fine-tuned training can shift models toward compositional responses or improve accuracy when prompts reassign label meanings.
By Minglai Yang, Xinyu Guo, Renxiong Wang, Sunjie Hou, Daniel Yue Zhang, Yunzhong He