MC‑EvoBench is a controlled Minecraft benchmark designed to audit experience‑driven self‑improvement in agents. It features 19 evolution chains (12 primary, 7 calibration/probe) and three graph‑aligned probe families, each task represented as a task‑state graph with stage‑level verifiers and strict source‑target boundaries. The benchmark evaluates probes under no‑prior, related, unrelated, and shuffled source conditions, reporting metrics such as TSR, WGP, and VMP while preserving provenance records.
arXiv:2608. 07545v1 Announce Type: cross Abstract: An LLM agent's capability depends not only on model weights but on its harness: prompts, tools, skills, and control flow.
EVOHARNESSBENCH is a new benchmark that tests how LLM-based agents handle changes in their tool, skill, and agent harnesses over time. It includes 17 deterministic harness streams with 802 tasks, 520 tools, 42 skills, and 62 agents, and evaluates agents in two settings: deployment evaluation and self‑evolving adaptation evaluation. The study finds that harness expansion can cause forgetting, adaptation gains are inconsistent, and preserving old competence does not always aid new capability adaptation, highlighting harness evolution as a distinct challenge for agent development.
arXiv:2609.04280v2 Announce Type: replace-cross Abstract: Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what the...
arXiv:2609.24663v1 Announce Type: new Abstract: Self-evolving agents convert interaction feedback into persistent artifacts, such as memories or skills, which in turn guide subsequent decisions. As t...
arXiv:2608. 08466v1 Announce Type: new Abstract: Modern LLM agents are often improved by modifying prompts, tools, or workflows manually, while the executable scaffold surrounding the model---the \emph{harness}---is typically treated as a fixed artifact after deployment.
MineExplorer is a benchmark designed to assess the open‑world exploration abilities of multimodal large language models (MLLMs) in Minecraft. It filters out tasks that rely heavily on Minecraft‑specific knowledge, organizes tasks into ReAct‑style capabilities, and composes atomic tasks into implicit multi‑hop challenges. A multi‑agent synthesis workflow creates reliable task graphs, sandbox scenes, and rule‑based milestone evaluators, and human evaluation confirms its superiority over a single‑agent baseline. Experiments show that while advanced MLLMs can handle many single‑hop tasks, they struggle with longer trajectories that require coordinating hidden prerequisites, and larger models or different thinking modes do not consistently improve performance.
The paper introduces environment‑probing curation, a deployment‑compatible method that equips asynchronous curator agents with read‑only world tools to verify, scope, and refresh candidate memories without retraining models. In a GitHub Copilot‑based harness, this approach improves pass rates on CLBench from 39% to 73%, boosts reward metrics, and reduces both query counts and task‑agent costs. Across six APEX management‑consulting tasks, the method consistently outperforms baselines, yielding higher rewards and fewer tool calls while maintaining a compact task‑time interface.
arXiv:2605. 27898v2 Announce Type: replace Abstract: As LLMs are increasingly deployed as agents, reliable assessment of their agentic capabilities has become essential.
The paper introduces The Trinity, a recursive self‑improvement framework for multimodal agents that couples world generation with harness evolution through three roles: Creator, Explorer, and Evolver. By shifting task organization from surface categories to shared capability requirements, it builds a hierarchical capability tree and a reusable harness tree, enabling efficient transfer learning across heterogeneous tasks. Experiments show the evolved harnesses improve mean performance by 34 points on 28 training benchmarks and 26 points on 18 unseen benchmarks, including two real‑world embodied tasks.
Mingbird is a local‑first agent harness designed for small open‑weight language models (2–9 B) that run on ordinary laptops. It introduces ten mechanisms—such as a byte‑level net‑zero prefill budget, a finish gate that re‑reads the task before accepting completion, and signature‑level loop detection—to address common failure modes that arise from the harness rather than the model itself. In controlled experiments on the LRAB benchmark and the $ au^2$‑bench, Mingbird achieves higher overall scores (0.886 and 0.856 respectively) compared to other harnesses, and its ablation studies show that each mechanism contributes measurable performance gains.
arXiv:2610.14146v1 Announce Type: new Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its fi...
VeriEvolve is an evaluator-first evolution framework that first builds an executable evaluator from task instructions, then uses this evaluator as a frozen fitness signal while evolving agents. The system generates challenge states and counterfactual test cases to refine the evaluator across rounds, producing 30 evaluators for 30 MCPMark Filesystem tasks with perfect agreement on generated suites and high agreement on held-out states. Evaluator-guided evolution significantly improves externally verified success rates for GLM-5.2 and Claude, and cross-environment studies reveal complementary limits from observability, state representation, and search/execution capacity.