arXiv AI
Jul 15
Rethinking the Evaluation of Harness Evolution for Agents
arXiv:2607. 12227v1 Announce Type: new Abstract: We revisit the evaluation of automatic harness evolution for LLM agents.
Harness-Delta Attribution: Dissecting Gains From Harness Evolution
The paper introduces Harness-Delta Attribution (HDA), a protocol that breaks down the performance gains from harness evolution into overfitting, test-time scaling, and residual generalizable improvement. Applying HDA to 11 executor models and three harness-evolution frameworks across various tasks, the study finds that over half of the search-set gains are due to overfitting or scaling, with limited transfer to held-out tasks. The results highlight the need for attribution-aware evaluation in future harness evolution research.
Related stories
arXiv Computation and Language
Sep 2
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
arXiv:2609.01437v1 Announce Type: cross Abstract: As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly...
arXiv Machine Learning
1d ago
Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization
The paper investigates how optimizing the harness of language model agents—without changing model weights—affects performance. It distinguishes between gains that improve success on tasks already within reach (elicitation) and gains that broaden the range of solvable tasks (expansion). By comparing optimized harnesses to their pre‑optimization versions across static and dynamic tasks, the study finds that benefits can reverse at higher attempt budgets and that aggregate gains may mask shortcomings on harder tasks, highlighting the need for detailed diagnostics.
arXiv AI
Oct 1
Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer
The paper introduces a self‑evolving harness framework where a frozen language‑model agent first solves tasks and then edits its own harness based on run records. Using a 49‑line seed harness, the evolved harness improves average scores on in‑distribution benchmarks by 4.48 points and on out‑of‑distribution benchmarks by 12.64 points, surpassing Codex on the former and matching it on the latter. Continued evolution on a specific out‑of‑distribution benchmark further raises performance, and the study analyzes emergent mechanisms such as output truncation and history compaction.
arXiv AI
Aug 7
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
arXiv:2608. 06301v1 Announce Type: new Abstract: As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them.
arXiv AI
Aug 21
Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection
arXiv:2608. 20169v1 Announce Type: cross Abstract: We present a novel approach to efficient LLM agent harness optimization through adaptive validation task selection.
arXiv Computation and Language
Sep 7
EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?
EVOHARNESSBENCH is a new benchmark that tests how LLM-based agents handle changes in their tool, skill, and agent harnesses over time. It includes 17 deterministic harness streams with 802 tasks, 520 tools, 42 skills, and 62 agents, and evaluates agents in two settings: deployment evaluation and self‑evolving adaptation evaluation. The study finds that harness expansion can cause forgetting, adaptation gains are inconsistent, and preserving old competence does not always aid new capability adaptation, highlighting harness evolution as a distinct challenge for agent development.
arXiv Computation and Language
Sep 14
EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?
arXiv:2609.04280v2 Announce Type: replace-cross Abstract: Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what the...
arXiv AI
Sep 10
Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails
arXiv:2609.09134v1 Announce Type: new Abstract: Agent harnesses (the system prompt, tool set, execution hooks, and context-management scaffolding around a model) are a critical determinant of agentic...
Hugging Face Trending Papers
Aug 10
Evo-Bench: Can Language Models Improve Agent Harness?
Large Language Models (LLMs) have driven rapid progress in autonomous agents, yet standard evaluations remain confined to static task solving. An emerging frontier is harness evolution---the agent's capacity to autonomously optimize its own operating harness.
arXiv AI
Jul 9
Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1
arXiv:2607. 06764v1 Announce Type: new Abstract: Recent progress on ARC-AGI-1 from disclosed architectures has come broadly from two regimes: heavy test-time compute over frontier models (evolutionary search, exhaustive sampling, extended chain-of-thought), or benchmark-specific training in which small models are fine-tuned on ARC data, often with task-specialized architectures.
arXiv Machine Learning
Jul 30
DREvo: Distilling Recalibrated Historical Experience for Harness Self-Evolution
arXiv:2607. 26722v1 Announce Type: cross Abstract: Harness plays a critical role in large language model agent performance, and building a high-performing harness requires substantial expert effort.