← Back to all news
arXiv Computation and Language By Mingyu Lu, Timothy Ossowski, Tristan Naumann, Hoifung Poon, Peniel Argaw

HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation

Read the original on arXiv Computation and Language →

HERO is a method that improves large language models for program generation by combining offline and test‑time harness optimization. It learns reusable prompt updates from execution‑verified successes and failures, validates candidate revisions before adoption, and then refines each generated code instance using execution feedback. On four SQL, Cypher, and Python benchmarks, HERO achieves the best offline performance, raising pass@1 by 7.6 points over the initial prompt and adding another 2.0 points through test‑time refinement while improving execution efficiency.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Jul 10

TTHE: Test-Time Harness Evolution

arXiv:2607. 08124v1 Announce Type: cross Abstract: The behavior of an LLM agent is determined not only by the underlying model, but also by its harness: the executable program that constructs context, invokes tools, verifies intermediate results, and recovers from failures.

By Jun Nie, Yonggang Zhang, Jun Song, Qianshu Cai, Dahai Yu, Yike Guo, Xinmei Tian, Bo Han
llmsagentsdata-engineering
More like this →
arXiv AI
Sep 24

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.

By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati
llmsbenchmarks
More like this →