The paper investigates how rewording questions affects the relative performance of language models and whether providing models with tools can stabilize these comparisons. Using a matched protocol that isolates answer readout, reasoning, tool calls, and reporting, the authors evaluate six open models across five domains and five question wordings. They find that tool usage can increase accuracy for programmatic answers but also tends to widen wording sensitivity, especially when call frequency varies with wording and depends on the interface and instructions.
By Mateusz Modrzejewski, Tomasz Radzikowski
The paper introduces a fine-grained method called interactions to analyze prompt sensitivity in large language models (LLMs). By decomposing output scores into nonlinear interactions, the authors show that subtle prompt changes can destabilize these interactions even when overall outputs stay unchanged. They propose an Interaction-based Prompt Sensitivity (IPS) metric and use it to evaluate 50 open-source LLMs, finding that supervised fine‑tuning, larger model scales, dense architectures, and few‑shot learning all reduce prompt sensitivity, primarily by stabilizing low‑order interactions.
By Ruiyang Qin, Qingzhuo Wang, Tian Wang, Zhihua Wei, Wen Shen
Recent advancements in Large Language Models (LLMs) have enabled sophisticated reasoning and content generation, yet their inherent stochasticity poses significant challenges for ensuring predictive credibility. While traditional uncertainty taxonomy paradigms, such as the dichotomy of aleatoric and epistemic uncertainties, provide conceptual foundations, they often fail to capture the multi-component and multi-stage nature of LLM generation and struggle to evaluate the effectiveness of various Uncertainty Quantification (UQ) methods.
arXiv:2604.11662v2 Announce Type: replace
Abstract: Recent work has shown that the hidden states of large language models contain signals useful for uncertainty estimation, motivating a growing inter...
By Joe Stacey, Hadas Orgad, Kentaro Inui, Benjamin Heinzerling, Nafise Sadat Moosavi
arXiv:2605.30018v3 Announce Type: replace-cross
Abstract: Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of the...
By Tanmoy Chakraborty, Ayan Sengupta, Suparna Bhattacharya, Partha Pratim Chakrabarti, Amlan Chakrabarti, Supratik Chakraborty, Partha Pratim Das, Lipika Dey, Richa Singh, Mayank Vatsa
The paper investigates how small lexical changes in prompts can cause large performance swings in large language models. Using a dataset of 132,000 prompt variants, the authors uncover a scaling law linking higher average task performance to lower variance and greater robustness. They identify domain-specific terminology and explicit action directives as key linguistic factors that stabilize prompts, and propose an automated Prompt-Refining Agent that reduces performance variance by 40.7% in code generation while maintaining or improving mean performance.
By Qipeng Xie, Zi Liang, Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, Haiqin Yang, Kaishun Wu
The paper introduces perturber, a stateless service and Python library designed to apply single, semantics‑preserving text perturbations. It offers a catalog of 68 perturbations across 11 families, enabling reproducible, isolated application of each perturbation without coupling to a specific benchmarking stack. The authors evaluate the tool’s correctness, reproducibility, and cost, and provide a live playground and introspectable catalog for benchmark selection.
By Khoi-Nguyen Tran
arXiv:2605.28190v2 Announce Type: replace
Abstract: Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embeddin...
By Manuel Frank, Haithem Afli
arXiv:2610.01428v1 Announce Type: cross
Abstract: Generalization in large language models (LLMs) is the ability to produce consistent and semantically stable outputs when the same input is expressed...
By Nagham Omar, Mahmoud Jabarin, Maya Rozenshtein, Rom Himelstein, Avi Mendelson, Amit LeVi
As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy.
The paper introduces Prompt-agnostic Language Models, focusing on train‑time techniques to embed prompt robustness during instruction tuning. It shows that mixing paraphrases in training causes significant gradient interference—57% of per‑paraphrasing gradients conflict—leading to the development of two gradient‑merging methods, Prompt‑TIES and Prompt‑DARE, which resolve this interference. Experiments across four models demonstrate that these methods substantially reduce per‑example variance and improve local robustness with minimal impact on overall accuracy.
By Frederic Sadrieh
The paper introduces the "Prompt Lottery" phenomenon, where paraphrastic rephrasings of judge prompts in LLM-as-a-Judge (LLMaJ) lead to inconsistent judgments and can even reverse model rankings. A large-scale study across various tasks, judge families, and model backbones demonstrates that these prompt variations significantly alter evaluation outcomes. A user study shows that LLMaJ judges are 6–9 times more sensitive to prompt rephrasings than humans, and that aggregating multiple prompts improves robustness by focusing on consistent judgments.
By Ilija Subasic, Brett Levert, Andrew Rabinovich, Sajjad Pakdamansavoji, Zhao Chen