The paper introduces BudgetAPO, a single‑stage prompt optimizer designed for tight‑budget scenarios where only a few hundred model calls are feasible. BudgetAPO adapts the evaluation slice size to each task’s noise level, uses a fixed slice to turn accept/reject decisions into paired comparisons, and applies a reflective operator to jointly rewrite reasoning strategy and output format. Across seven benchmarks and five subject models, BudgetAPO outperforms existing methods, ranking first on every subject and requiring far fewer calls to achieve comparable or superior performance.
arXiv:2609.39927v1 Announce Type: new Abstract: Prompt optimization improves the performance of language-model systems on downstream tasks by refining their prompts. Classical methods evaluate prompt...
arXiv:2608. 10471v1 Announce Type: new Abstract: Prompt optimizers automate the search for prompts that improve language-model performance, but existing methods rely on a predefined optimization procedure: the algorithm determines which candidates to explore and how the search progresses, while the language model generates or refines prompt proposals.
The paper investigates how test‑time computation can enhance language models and at what cost, introducing the SELF‑POT benchmark to evaluate this across competition mathematics, competitive programming, and agentic workflows. SELF‑POT separates candidate coverage from final accuracy, tracks correctness transitions under revision, and measures protocol completion alongside task success. Using a unified budget rule, the study compares direct inference, parallel sampling, and self‑revision across five low‑cost reasoning models, revealing that selection rules and failure handling significantly influence gains and cost savings.
arXiv:2606. 04661v1 Announce Type: cross Abstract: Prompts tuned for accuracy often grow long, raising inference cost on every model call.
arXiv:2608. 04336v1 Announce Type: cross Abstract: Code generation systems make each LLM call with a model, a prompt, and decoding settings.
arXiv:2610. 00416v1 Announce Type: new Abstract: Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamination, and assessing harder tasks can require expensive grading or execution infrastructure.
The paper investigates why prompt optimization works better for some tasks than others by decomposing reward variance into response variance and system‑prompt variance. It finds that optimization succeeds when system‑prompt variance dominates, and that adding more user prompts can actually reduce this variance, especially on heterogeneous datasets. To address this, the authors propose $p1$, a filtering method that selects a small set of high‑variance user prompts, which improves optimization on reasoning benchmarks and even allows a system prompt trained on just two AIME 24 prompts to generalize well.
The paper introduces the Expected Decoding Rounds (EDR) objective, a theoretical framework that models speculative decoding as a Markov reward process and directly optimizes the number of decoding rounds. EDR weights local rejection costs by state occupancies, eliminating auxiliary hyperparameters and enabling unbiased stochastic optimization via temporal‑difference gradients. Applying EDR to fine‑tune state‑of‑the‑art draft models (DSpark and DFly) improves mean accepted length and outperforms prior training objectives across nine benchmarks in math reasoning, code generation, and chat.
Semantic Feature Analysis (SFA) is a method that refines agent specifications without performing any rollout-based search. It analyzes existing execution traces, clusters workflow node outputs, extracts semantic feature classes via an extended subject‑verb‑object schema, ranks these features with a decision tree, and injects the most impactful features back into the system prompt. Evaluations on four benchmarks show that SFA consistently outperforms five prompt‑optimisation algorithms and a single‑reflection baseline, especially when rollout costs are high.
arXiv:2608. 12150v1 Announce Type: new Abstract: Standard evaluation of large language models assumes stable model rankings across inference conditions.
The paper introduces AdaptiveSpec, a training‑free speculative decoding method that simultaneously adapts the per‑step verification rule and the draft‑tree shape using signals generated during decoding. It replaces the fixed token‑match rule with a margin‑based threshold and adjusts tree depth, width, and node count based on draft confidence and recent acceptance history, allowing the total draft count to vary. Experiments on SGLang show up to 56% throughput gains over EAGLE‑3 while maintaining 93% of lossless task accuracy on GSM8K, MATH‑500, and HumanEval across three models.
The paper demonstrates that the number of candidates generated during test-time scaling of large language models does not fully capture the system cost. By comparing different generation schedules (e.g., one batched call versus multiple serial calls) while keeping the total candidate count fixed, the authors show that serial calls consume significantly more GPU energy and latency. The study suggests that reporting candidate count alone is insufficient; evaluations should also include generation schedule and GPU-level metrics.