The study investigates whether large language models (LLMs) can automatically improve the harnesses used for hardware design verification. By evolving harnesses around a fixed subject model on 12 proprietary root‑cause localization tasks, the researchers found that automatically evolved harnesses increased completed attempts by 71‑76% and task coverage by 80‑100%, though overall correct attempts improved only 18‑24%. Despite these gains, the evolved harnesses did not consistently consolidate into a single dominant solution across tasks and metrics, and a separate cross‑benchmark case showed that a repair harness could yield a 35.6% increase in functional passes over a baseline.
Large Language Models (LLMs) have driven rapid progress in autonomous agents, yet standard evaluations remain confined to static task solving. An emerging frontier is harness evolution---the agent's capacity to autonomously optimize its own operating harness.
Ecdysis is a framework for training runtime harnesses for large language model agents more efficiently. It distinguishes between model‑specific issues and systematic harness deficiencies by aggregating failures across multiple task instances and uses Failure‑Driven Collaborative Refinement to diagnose and fix harness problems. The approach reduces training time by up to 1.84× and improves harness reasoning accuracy by 18.56%.
arXiv:2608. 20169v1 Announce Type: cross Abstract: We present a novel approach to efficient LLM agent harness optimization through adaptive validation task selection.
Ecdysis is a framework for training runtime harnesses for large language model agents that reduces training time and improves performance. It distinguishes between model‑specific issues and systematic harness deficiencies by aggregating failures across multiple task instances and uses Failure‑Driven Collaborative Refinement to diagnose and correct harness problems. Experiments show up to a 1.84× speedup in harness training and an 18.56% increase in reasoning accuracy.
arXiv:2607. 26598v1 Announce Type: cross Abstract: Large language model (LLM) agents may recover from a failure within an episode or after a retry, yet the same execution failure can recur in later tasks because post-episode feedback rarely revises the persistent harness that guides future interactions.
arXiv:2608. 01918v1 Announce Type: new Abstract: Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments.
arXiv:2607. 12227v1 Announce Type: new Abstract: We revisit the evaluation of automatic harness evolution for LLM agents.
arXiv:2608. 06153v1 Announce Type: cross Abstract: Automated skill evolution enables Large Language Model (LLM) agents to continuously improve without expensive retraining.
The paper introduces a self‑evolving harness framework where a frozen language‑model agent first solves tasks and then edits its own harness based on run records. Using a 49‑line seed harness, the evolved harness improves average scores on in‑distribution benchmarks by 4.48 points and on out‑of‑distribution benchmarks by 12.64 points, surpassing Codex on the former and matching it on the latter. Continued evolution on a specific out‑of‑distribution benchmark further raises performance, and the study analyzes emergent mechanisms such as output truncation and history compaction.
ChipMEM introduces a verification‑grounded memory layer for electronic design automation agents that combines cross‑task procedural memory with within‑trajectory statistical guidance. The procedural component stores a skill only after it passes synthesis, simulation, or formal checks, while a Bayesian component ranks recovery strategies based on tool‑call outcomes. Experiments on RTLRewriter‑Bench and CVDP tasks show that ChipMEM improves equivalence‑passing outputs and area metrics compared to agents without memory.
arXiv:2607. 08124v1 Announce Type: cross Abstract: The behavior of an LLM agent is determined not only by the underlying model, but also by its harness: the executable program that constructs context, invokes tools, verifies intermediate results, and recovers from failures.
arXiv:2609.01437v1 Announce Type: cross Abstract: As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly...