arXiv AI
Auditing LLM Program Search for Bermudan Stopping with Primal--Dual Certificates
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 29
When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Model
arXiv:2606. 21641v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have been proposed as hyperparameter-optimization (HPO) advisors that "warm-start" search from prior knowledge, proposing strong configurations in very few evaluations.
arXiv Machine Learning
Oct 8
The Winner's Curse in LLM Self-Improvement Loops: Selection Noise, Lock-in, and Acceptance Rules
The paper investigates how self‑improving large language models (LLMs) can suffer from the winner’s curse when selecting better candidates based on noisy evaluation sets. By modeling correlated errors and empirically testing Qwen models that rewrite their own instructions, the authors find that most proposals after the first are harmful and that the reported gains on the selection set often overstate true held‑out performance. The study shows that larger selection sets can improve held‑out gains on some tasks (e.g., TREC) but not others (e.g., GSM8K), and that common acceptance rules do not outperform simple greedy selection across entire runs.
arXiv Machine Learning
Jul 1
Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by Language Models
arXiv:2606. 31630v1 Announce Type: new Abstract: Language models increasingly write probabilistic programs (in NumPyro, Stan, or Pyro), but a program that compiles, runs, and passes every unit test can still be \emph{statistically} wrong -- a Gaussian likelihood for heavy-tailed data, a Poisson for over-dispersed counts, an invalid prior support, or a pathological parameterization.
arXiv Computation and Language
Sep 25
StepCOPS: Closed-Testing Lower-Tail Certificates for Language-Model Policy Selection
StepCOPS is a new method for selecting a language‑model policy from many checkpoints, prompts, and decoding rules by providing closed‑testing lower‑tail certificates. It uses an independent proposal split to nominate a lower‑tail floor for each candidate, applies exact binomial tests on a fresh certification split, and employs Holm’s step‑down procedure to certify a set of floors. In experiments across 24 configurations and 11 benchmarks, StepCOPS achieves 96.4% selected‑policy coverage, raises the certified floor by 1.5 points over prior methods, stays 0.6 points below a large‑reference jury oracle, and abstains in 2.4% of trials.
arXiv Statistics ML
Oct 28
Confidence Is Not a Promise: A Reference Audit of Typed Decision Models
The paper evaluates how well typed decision models, which output probability distributions, actually provide reliable guarantees. By comparing open-weight Laya and Jev 1.13 across 190 evaluations, the study finds that point‑estimate thresholds frequently break promised accuracy, while a Beta lower‑bound rule and Learn‑then‑Test approach maintain guarantees more consistently. The work also explores calibration across languages, the impact of noise, data requirements for fine‑tuning, and the limitations of confidence in predicting outcomes or providing faithful explanations.
arXiv AI
Aug 26
When May an Agent Stop? Evidence-Carrying Termination for Tool-Using LLMs
The paper introduces Evidence-Carrying Termination (ECT), a method that allows tool‑using large language model agents to declare a task complete only when a typed certificate links every required answer claim to valid, in‑scope trace evidence and a deterministic replay confirms the claimed value. In controlled experiments across 48 synthetic tasks and 576 trajectories, ECT eliminated unsafe completions and premature unsupported terminations, outperforming existing termination critics and controllers while maintaining comparable supported completion rates.