The paper proposes a two‑stage framework for expanding and refining a set of prompts for language models under a limited evaluation budget. In the first stage, it generates a finite candidate set of prompts represented by moderate‑dimensional vectors. The second stage builds a surrogate model of prompt scores and sequentially selects prompts for evaluation, proving consistency of this procedure and introducing a refinement step that searches beyond the initial candidates. Numerical experiments show the method outperforms baseline prompt selection, especially when evaluation calls are constrained.
arXiv:2609.39927v1 Announce Type: new Abstract: Prompt optimization improves the performance of language-model systems on downstream tasks by refining their prompts. Classical methods evaluate prompt...
The paper investigates how small lexical changes in prompts can cause large performance swings in large language models. Using a dataset of 132,000 prompt variants, the authors uncover a scaling law linking higher average task performance to lower variance and greater robustness. They identify domain-specific terminology and explicit action directives as key linguistic factors that stabilize prompts, and propose an automated Prompt-Refining Agent that reduces performance variance by 40.7% in code generation while maintaining or improving mean performance.
GFlowPO introduces a probabilistic framework for optimizing language model prompts by treating prompt search as posterior inference over latent prompts, regularized by a meta-prompted reference-LM prior. The method first fine‑tunes a lightweight prompt‑LM using an off‑policy Generative Flow Network objective that reuses past prompt evaluations for sample‑efficient exploration. It then applies a training‑free Dynamic Memory Update to refine the meta‑prompt with diverse and top‑performing prompts, leading to consistent improvements over recent discrete prompt optimization baselines on few‑shot classification, instruction induction, and question‑answering tasks.
The paper introduces soft‑prompt tuning as a lightweight, automated method for evaluating large language models (LLMs). By optimizing only ten soft‑prompt vectors—about 0.0006% of a 7B model’s parameters—within 150 steps, the approach adapts models to specific benchmark formats, improving format compliance while preserving knowledge accuracy. Experiments across seven models and eleven datasets show that this method matches strong evaluation‑harness baselines, enhances evaluation signals on challenging datasets, and yields cleaner outputs that align better with exact‑match scoring and independent LLM judges.
The paper introduces the "Prompt Lottery" phenomenon, where paraphrastic rephrasings of judge prompts in LLM-as-a-Judge (LLMaJ) lead to inconsistent judgments and can even reverse model rankings. A large-scale study across various tasks, judge families, and model backbones demonstrates that these prompt variations significantly alter evaluation outcomes. A user study shows that LLMaJ judges are 6–9 times more sensitive to prompt rephrasings than humans, and that aggregating multiple prompts improves robustness by focusing on consistent judgments.
arXiv:2607. 28657v1 Announce Type: new Abstract: Large Language Models (LLMs) often require carefully crafted prompts to unlock their full potential, which can be a barrier for non-expert users.
RECAST is a new framework that generates datasets with far more constraints per example than existing benchmarks, aiming to push large language models (LLMs) to better follow complex instructions. The authors built RECAST-30K, a 30,000‑instance dataset covering 19 constraint types extracted from real prompt‑response pairs, and showed that fine‑tuning on it improves LLMs’ ability to handle complex tasks without harming general performance. RECAST also provides rule‑based and LLM‑based validators for automatic constraint verification, enabling reward‑based reinforcement learning to further enhance model performance on challenging tasks.
The paper introduces OR‑Clarify, a benchmark that tests whether large language models can identify missing elements in natural‑language optimization requests before formulating a mathematical model. Each task provides a partial problem description and hides structured slots; agents interact with a simulated user to recover these slots, with metrics for accuracy, stopping decisions, and interaction cost. The authors also propose InterOPT, a two‑stage framework that detects unresolved gaps and decides whether to ask further questions or stop, achieving superior slot recovery in choice‑based experiments and competitive performance in open‑ended settings.
arXiv:2511. 19829v3 Announce Type: replace Abstract: Prompt optimization has become a central mechanism for eliciting strong performance from LLMs, and recent work has made substantial progress by proposing diverse prompt evaluation metrics and optimization strategies.
The paper introduces a method that represents language models as log‑likelihood vectors over prompt‑response pairs, enabling the construction of model maps that compare conditional distributions. Squared Euclidean distances in this vector space approximate KL divergence, and experiments show that these maps reveal global structure related to model attributes and task performance. The approach also captures systematic shifts from prompt changes, supports additive compositionality for predicting downstream scores, and offers PMI vectors to mitigate unconditional distribution effects, thereby aiding analysis and prediction of input‑dependent behavior.
Evaluation of large language models (LLMs) remains largely pre-scientific: it involves many design decisions, with little guidance on how to make them. These choices can substantially affect measured...
arXiv:2606. 24841v1 Announce Type: new Abstract: Prompt-based learning has emerged as a dominant paradigm in natural language processing.