DeltaSelect: Affordable A/B Testing for Coding Agents

DeltaSelect is an open‑source method that selects a small, cost‑effective set of tasks for A/B testing coding agents, using Pearson correlation to identify tasks whose single‑run results reliably reflect full‑benchmark performance. It maps fractional verifier outputs to a common score via linear regression and operates within a specified dollar budget, making it suitable for repeated baseline‑versus‑candidate comparisons during development rather than for model ranking. In a case study with gpt‑5.6‑luna, DeltaSelect reduced costs by 58.1% and improved calibrated scores across 13 evaluations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• agents
• benchmarks