← Back to all news
arXiv AI

Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI

The paper introduces a method for a Builder model to learn meta-skills—principles that guide when and how to provide support—to improve the execution environments of a Target model during test time. By training on feedback from the Target on a development set, the Builder constructs harnesses for new tasks using a frozen skill bank. Experiments on Harness-Bench and NewtonBench show that using the full meta-skill bank boosts macro-average performance by 8.95 percentage points compared to no-skill construction and by 12.02 points over directly delivering the same skill bank to the Target.

Related stories

arXiv AI
Aug 20

SPADE: Self-Play in Adaptive Synthetic Executable Environments

SPADE (Self-Play in Adaptive Synthetic Executable Environments) is a reinforcement‑learning framework where a single large language model acts as both an Environment Designer—creating executable, long‑horizon training environments—and a Reasoning Agent—learning to act within those environments. The framework uses a regret signal based on the difference between rewarded performance with and without privileged hints to guide the Designer toward environments that are challenging yet solvable. Experiments show that, when scaled to 30‑billion‑parameter models, SPADE outperforms fixed‑environment baselines by significant margins across math, science, code, and reasoning benchmarks, and improves tool‑use performance on BFCL‑v4 and ACEBench‑Agent. whyItMatters":"By making environment design a learnable component, SPADE enables continuous self‑improvement and demonstrates that adaptive, self‑generated training environments can substantially boost language‑model performance across diverse tasks."

By Bo Liu, Simon Yu, Yiding Jiang, Ao Qu, Andrew Zhao, Zichen Liu, Junsu Kim, Zijian Zhou, Seungone Kim, Tongzheng Ren, Mickel Liu, Hanfei Yu, Zhaorun Chen, Weiyan Shi, Paul Pu Liang, Luke Zettlemoyer, Yejin Choi, Natasha Jaques
llmsagentsreinforcement-learningbenchmarks
More like this →
arXiv AI
Oct 9

Agentic-TTT: Training test-time policy for test-time training

Agentic‑TTT introduces a test‑time policy that decides when and how to apply test‑time training (TTT) to large language models. By treating TTT procedures as tools and learning from the utility gains of its decisions, the policy can trade off performance improvements against computational cost. On a benchmark, Agentic‑TTT nearly doubles the utility of the base model, adapts to unseen domains, and demonstrates autonomous self‑improvement capabilities.

By Jiahao Lu, Mohan Kankanhalli
llmsagentsbenchmarks
More like this →
arXiv Computation and Language
Sep 2

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

arXiv:2609.01437v1 Announce Type: cross Abstract: As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly...

By Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He, Chen Huang, Maojia Song, Zhiyuan Zeng, Shaowen Wang, Jinkai Liu, Yunfeng Shi, Jiaheng Liu, Shen Yan, Wenhao Huang, Ge Zhang, Wenxuan Zhang
llmsagentsbenchmarks
More like this →
arXiv Computation and Language
Sep 7

EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?

EVOHARNESSBENCH is a new benchmark that tests how LLM-based agents handle changes in their tool, skill, and agent harnesses over time. It includes 17 deterministic harness streams with 802 tasks, 520 tools, 42 skills, and 62 agents, and evaluates agents in two settings: deployment evaluation and self‑evolving adaptation evaluation. The study finds that harness expansion can cause forgetting, adaptation gains are inconsistent, and preserving old competence does not always aid new capability adaptation, highlighting harness evolution as a distinct challenge for agent development.

By Zixuan Ke, Vaidehi Patil, Haizhou Shi, Yang Li, Ye Liu, Sarath Shekkizhar, Anurag Koul, Jiayu Wang, Xuan Phi Nguyen, Semih Yavuz, Mohit Bansal, Shafiq Joty
llmsagentsbenchmarks
More like this →