arXiv:2608. 10471v1 Announce Type: new Abstract: Prompt optimizers automate the search for prompts that improve language-model performance, but existing methods rely on a predefined optimization procedure: the algorithm determines which candidates to explore and how the search progresses, while the language model generates or refines prompt proposals.
The study compares two ways of obtaining predictions from language models fine‑tuned on customer behavior: scoring answer tokens directly versus generating a written rationale and then scoring the resulting answer. Across 13 model‑domain cells covering four retail tasks, scored readouts consistently rank outcomes more accurately than generated readouts, with an AUC improvement ranging from 1.5 to 14.5 points. The authors also find that a third readout—eliciting a probability before any verdict—improves calibration but only when outcome rates are represented in training, and they recommend using generated rationales for interpretability while relying on scored heads for ranking.
The paper argues that verbalized confidence—once viewed as overconfident and coarse—has become the preferred soft‑scoring method for LLM‑as‑a‑Judge on top‑tier proprietary models released after 2025. Experiments on SummEval, AggreFact, and HelpSteer2 across up to 18 LLMs show that log‑probabilities are no longer the best signal, and that adding an overconfidence advisory and self‑debate further improves calibration and robustness. The authors note that these enhancements incur little accuracy loss on post‑2025 models but do affect pre‑2025 ones, highlighting a compatibility shift in how confidence should be measured.
The paper introduces the Grounded Integration Measure (GIM), a benchmark of 820 expert‑authored problems designed to test models on tasks that integrate multiple cognitive operations such as constraint satisfaction, state tracking, epistemic vigilance, and audience calibration. GIM emphasizes realistic, broadly accessible knowledge rather than specialized expertise, and uses a judge‑aware 2‑parameter logistic IRT model to produce robust ability estimates across 53 model‑thinking‑level configurations. The authors provide a comprehensive leaderboard of 22 models and 47 test configurations, and conduct an extensive study on how test‑time compute affects model capability, finding that configuration choices like thinking budget and quantization can be as influential as model selection itself. whyItMatters":"By focusing on integration of multiple cognitive domains, GIM offers a more realistic assessment of model reasoning capabilities than benchmarks that either overemphasize memorization or abstract reasoning alone."
The paper investigates the reliability of LLM judges in the Arena-Hard benchmark, showing that aggregate scores can mask significant variability in individual verdicts. By running GPT‑4.1 across 500 prompts under eight different judge configurations and repeating each setup, the authors find that small changes—whether in temperature, criteria, or instruction framing—can shift win/tie/loss outcomes on up to 28% of prompts and alter aggregate scores by several points. The study concludes that stable aggregate scores do not guarantee repeatable individual judgments, and recommends that evaluations report both aggregate results and prompt‑level agreement.
arXiv:2606. 30840v1 Announce Type: new Abstract: LLM agents are becoming central to information retrieval: they issue retrieval queries, synthesize answers, and increasingly serve as judges for IR evaluation.
Large reasoning models (LRMs) often improve math and coding performance, but their effect on instruction following is unclear. We study IFEval with Qwen3 models (1.
arXiv:2608. 11727v1 Announce Type: new Abstract: When a coding agent obeys a rule, it may simply have been going to do that anyway.
The paper investigates how self‑improving large language models (LLMs) can suffer from the winner’s curse when selecting better candidates based on noisy evaluation sets. By modeling correlated errors and empirically testing Qwen models that rewrite their own instructions, the authors find that most proposals after the first are harmful and that the reported gains on the selection set often overstate true held‑out performance. The study shows that larger selection sets can improve held‑out gains on some tasks (e.g., TREC) but not others (e.g., GSM8K), and that common acceptance rules do not outperform simple greedy selection across entire runs.
arXiv:2610.28522v1 Announce Type: new Abstract: Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make...
The study evaluates whether prompt optimizers that improve a target metric inadvertently violate other unscored requirements. Using GEPA, MIPROv2, and TextGrad on HotpotQA, SQuAD 2.0, and IFEval, the authors find that all three optimizers cause regressions in unscored aspects such as citation validity and output format, especially on HotpotQA. However, a post‑hoc analysis shows that many violations stem from early termination of responses under a 200‑token limit, and with a 1,024‑token cap the regressions largely disappear, indicating the side effects are narrower than expected.
ESPO (Error-Structured Prompt Optimization) addresses prompt bloat in evolutionary prompt optimizers by separating optimization into Diagnose, Propose, and Select phases. It clusters training errors, generates diverse candidates, and applies bootstrap stability selection, achieving a 3.76‑point accuracy gain over GEPA on seven NLP benchmarks while producing 47% shorter prompts. Cross‑model tests on four additional student models confirm ESPO’s superior average accuracy, notably improving Qwen3 GSM8K from 15.00% to 91.40%.