arXiv AI
Jul 15
Rethinking the Evaluation of Harness Evolution for Agents
arXiv:2607. 12227v1 Announce Type: new Abstract: We revisit the evaluation of automatic harness evolution for LLM agents.
Forging More Tasks into Better Harnesses: Agentic Harness Evolution via Successive Scale Racing
The paper introduces Successive Scale Racing (SSR), a framework that improves agentic harness evolution by adaptively selecting optimal evidence combinations during each evolution round. SSR constructs diverse evidence sets based on task quality and coverage, then efficiently races candidate harnesses under a limited compute budget. Across three benchmarks, SSR outperforms baselines, steadily raising pass rates as the task pool expands from 40 to 200 while cutting validation executions by 60%.
Related stories
arXiv AI
Oct 1
Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer
The paper introduces a self‑evolving harness framework where a frozen language‑model agent first solves tasks and then edits its own harness based on run records. Using a 49‑line seed harness, the evolved harness improves average scores on in‑distribution benchmarks by 4.48 points and on out‑of‑distribution benchmarks by 12.64 points, surpassing Codex on the former and matching it on the latter. Continued evolution on a specific out‑of‑distribution benchmark further raises performance, and the study analyzes emergent mechanisms such as output truncation and history compaction.
arXiv AI
Jul 17
Reward-Free Evolving Agents via Pairwise Validator
arXiv:2607. 14408v1 Announce Type: new Abstract: A self-evolving agentic loop repeatedly proposes a tweaked version of an agent (its prompt template or program) and accepts or rejects the change based on a per-iteration quality signal.
arXiv AI
Aug 28
Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification
The paper introduces HarnessLens, a budget‑aware framework that evolves language‑model agent harnesses by jointly exploring task spaces and user‑configurable components. It derives candidate modifications from execution trajectories and selectively verifies them on behavior‑relevant tasks using an attributable‑evidence gate, thereby avoiding wasteful rollouts on unrelated behaviors. Experiments on three harnesses and four benchmarks show that HarnessLens improves held‑out performance by 7.6‑13.6% while using less evaluation budget than existing methods.
arXiv AI
Jun 11
Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference
arXiv:2606. 05922v2 Announce Type: replace Abstract: AI agents rely on a harness of skills, tools, and workflows to solve complex problems.
arXiv AI
Aug 21
Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection
arXiv:2608. 20169v1 Announce Type: cross Abstract: We present a novel approach to efficient LLM agent harness optimization through adaptive validation task selection.
arXiv AI
Aug 7
SkillHEX: Improving Agent Skills via Hypothesis-Driven Autonomous Exploration and Exploitation
arXiv:2608. 05628v1 Announce Type: new Abstract: Although agent skills equip LLMs with reusable procedural knowledge, manual maintenance suffers from high costs, unscalability, and misalignment.
arXiv AI
1d ago
When the Critic Falls Behind: Co-Evolving Rubrics and Harnesses for Continual Agent Self-Improvement
arXiv:2610.36826v1 Announce Type: new Abstract: Autonomous agents increasingly depend on harnesses that govern interaction, memory, tools, and execution around large language models (LLMs). While rec...
arXiv AI
Oct 19
Harness-Delta Attribution: Dissecting Gains From Harness Evolution
The paper introduces Harness-Delta Attribution (HDA), a protocol that breaks down the performance gains from harness evolution into overfitting, test-time scaling, and residual generalizable improvement. Applying HDA to 11 executor models and three harness-evolution frameworks across various tasks, the study finds that over half of the search-set gains are due to overfitting or scaling, with limited transfer to held-out tasks. The results highlight the need for attribution-aware evaluation in future harness evolution research.
arXiv AI
1d ago
The View from Nowhere: Population-Centered Agent Evolution
arXiv:2610.36660v1 Announce Type: new Abstract: Self-improving AI systems can revise the designs that govern their own behavior. Yet existing approaches center improvement on selected parent agents a...
arXiv AI
Jun 2
Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams
arXiv:2606. 01770v1 Announce Type: cross Abstract: Auto-harness systems such as A-Evolve, GEPA, and Meta-Harness improve LLM agents by optimizing prompts, skills, tools, memories, and supporting infrastructure from execution feedback, but they are typically evaluated on fixed offline benchmarks.
arXiv AI
Aug 7
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
arXiv:2608. 06301v1 Announce Type: new Abstract: As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them.