The paper investigates how small lexical changes in prompts can cause large performance swings in large language models. Using a dataset of 132,000 prompt variants, the authors uncover a scaling law linking higher average task performance to lower variance and greater robustness. They identify domain-specific terminology and explicit action directives as key linguistic factors that stabilize prompts, and propose an automated Prompt-Refining Agent that reduces performance variance by 40.7% in code generation while maintaining or improving mean performance.
By Qipeng Xie, Zi Liang, Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, Haiqin Yang, Kaishun Wu
The paper investigates why prompt optimization works better for some tasks than others by decomposing reward variance into response variance and system‑prompt variance. It finds that optimization succeeds when system‑prompt variance dominates, and that adding more user prompts can actually reduce this variance, especially on heterogeneous datasets. To address this, the authors propose $p1$, a filtering method that selects a small set of high‑variance user prompts, which improves optimization on reasoning benchmarks and even allows a system prompt trained on just two AIME 24 prompts to generalize well.
By Zhaolin Gao (Sid), Yu (Sid), Wang, Bo Liu, Thorsten Joachims, Kiant\'e Brantley, Wen Sun
The paper introduces a fine-grained method called interactions to analyze prompt sensitivity in large language models (LLMs). By decomposing output scores into nonlinear interactions, the authors show that subtle prompt changes can destabilize these interactions even when overall outputs stay unchanged. They propose an Interaction-based Prompt Sensitivity (IPS) metric and use it to evaluate 50 open-source LLMs, finding that supervised fine‑tuning, larger model scales, dense architectures, and few‑shot learning all reduce prompt sensitivity, primarily by stabilizing low‑order interactions.
By Ruiyang Qin, Qingzhuo Wang, Tian Wang, Zhihua Wei, Wen Shen
arXiv:2609.39927v1 Announce Type: new
Abstract: Prompt optimization improves the performance of language-model systems on downstream tasks by refining their prompts. Classical methods evaluate prompt...
By Junyang Chen, Zecheng Wang, Jingbang Chen
The paper introduces Prompt-agnostic Language Models, focusing on train‑time techniques to embed prompt robustness during instruction tuning. It shows that mixing paraphrases in training causes significant gradient interference—57% of per‑paraphrasing gradients conflict—leading to the development of two gradient‑merging methods, Prompt‑TIES and Prompt‑DARE, which resolve this interference. Experiments across four models demonstrate that these methods substantially reduce per‑example variance and improve local robustness with minimal impact on overall accuracy.
By Frederic Sadrieh
The paper investigates how different prompt adaptation techniques—discrete prompt optimization, prompt tuning, and prefix tuning—affect the internal processing of dense and mixture-of-experts language models. It finds that the adaptation location determines how task information reaches the output: text prompts mainly influence response construction, input-level prompts modify content representations, and layer-wise prefixes exert direct control via attention. In MoE models, these differences also lead to distinct expert‑routing patterns, while generic representation‑level metrics fail to reliably distinguish effective adaptations from placebo effects.
By Mikhail Aleksandrov, Nail Bashirov, Maksim Ivanov, Andrew Tsukanov, Timofey Karyagin, Alexander Miasnikov, Gleb Molodtsov, Valentin Samokhin, Aleksandr Beznosikov
arXiv:2608. 16627v1 Announce Type: cross Abstract: Natural language explanations (NLEs) are increasingly used as inputs, for example, as few-shot rationales that influence model behavior in in-context learning (ICL).
By Mahdi Dhaini, Adam Dejl, Juraj Vladika, Volkan \"Ozer, Barbara Plank, Gjergji Kasneci
arXiv:2610.27724v1 Announce Type: new
Abstract: Dense linear probes assess task readability from language model (LM) representations, but do not directly reveal how many features suffice for predicti...
By Kaiyang Wan, Yuxia Wang
arXiv:2606. 05486v1 Announce Type: cross Abstract: Prompt ambiguity is a common source of failure in large language models, but is difficult to localize because it is a latent property of the prompt, while existing attribution methods are designed to explain observable outputs such as logits or generated tokens.
By Govind Ramesh, Yao Dou, Wei Xu
The paper investigates how finetuning vision‑language models (VLMs) on class‑imbalanced data can severely hurt recall for rare classes, sometimes below zero‑shot performance. It discovers that using a different descriptive prompt at inference—termed cross‑prompt inference—recovers rare‑class recall across diverse VLMs and datasets, though it may slightly affect frequent‑class recall. Analysis with Tuning Contribution shows that cross‑prompt inference reduces reliance on finetuning‑induced computation, shifting the model toward its pretrained knowledge, and that the benefit disappears when the finetuning contribution is minimized.
By Emanuel Ben Baruch, Sivan Biham, Alon Shoshan, Yochai Zvik, Avi Ben-Cohen
arXiv:2609.39929v1 Announce Type: cross
Abstract: Long-context failures of RoPE-based language models can arise from RoPE's intrinsic tradeoff between maintaining stable token preferences and disting...
By Yuyang Wu, Yufeng Du, Hao Peng
arXiv:2606. 05799v1 Announce Type: new Abstract: Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's {\em behavioral robustness} to irrelevant or misleading information.
By Mohammad Anas Jawad, Cornelia Caragea