arXiv:2609.39927v1 Announce Type: new
Abstract: Prompt optimization improves the performance of language-model systems on downstream tasks by refining their prompts. Classical methods evaluate prompt...
By Junyang Chen, Zecheng Wang, Jingbang Chen
The paper investigates how small lexical changes in prompts can cause large performance swings in large language models. Using a dataset of 132,000 prompt variants, the authors uncover a scaling law linking higher average task performance to lower variance and greater robustness. They identify domain-specific terminology and explicit action directives as key linguistic factors that stabilize prompts, and propose an automated Prompt-Refining Agent that reduces performance variance by 40.7% in code generation while maintaining or improving mean performance.
By Qipeng Xie, Zi Liang, Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, Haiqin Yang, Kaishun Wu
GFlowPO introduces a probabilistic framework for optimizing language model prompts by treating prompt search as posterior inference over latent prompts, regularized by a meta-prompted reference-LM prior. The method first fine‑tunes a lightweight prompt‑LM using an off‑policy Generative Flow Network objective that reuses past prompt evaluations for sample‑efficient exploration. It then applies a training‑free Dynamic Memory Update to refine the meta‑prompt with diverse and top‑performing prompts, leading to consistent improvements over recent discrete prompt optimization baselines on few‑shot classification, instruction induction, and question‑answering tasks.
By Junmo Cho, Suhan Kim, Sangjune An, Minsu Kim, Dong Bok Lee, Heejun Lee, Sung Ju Hwang, Hae Beom Lee
The paper introduces soft‑prompt tuning as a lightweight, automated method for evaluating large language models (LLMs). By optimizing only ten soft‑prompt vectors—about 0.0006% of a 7B model’s parameters—within 150 steps, the approach adapts models to specific benchmark formats, improving format compliance while preserving knowledge accuracy. Experiments across seven models and eleven datasets show that this method matches strong evaluation‑harness baselines, enhances evaluation signals on challenging datasets, and yields cleaner outputs that align better with exact‑match scoring and independent LLM judges.
By Selen Erkan, Bastian Boll, Kristian Kersting, Bj\"orn Deiseroth, Letitia Parcalabescu
The paper introduces the "Prompt Lottery" phenomenon, where paraphrastic rephrasings of judge prompts in LLM-as-a-Judge (LLMaJ) lead to inconsistent judgments and can even reverse model rankings. A large-scale study across various tasks, judge families, and model backbones demonstrates that these prompt variations significantly alter evaluation outcomes. A user study shows that LLMaJ judges are 6–9 times more sensitive to prompt rephrasings than humans, and that aggregating multiple prompts improves robustness by focusing on consistent judgments.
By Ilija Subasic, Brett Levert, Andrew Rabinovich, Sajjad Pakdamansavoji, Zhao Chen
arXiv:2607. 28657v1 Announce Type: new Abstract: Large Language Models (LLMs) often require carefully crafted prompts to unlock their full potential, which can be a barrier for non-expert users.
By Oliver Savolainen, Emanuele Bastianelli, Hosein Azarbonyad