arXiv:2608. 06301v1 Announce Type: new Abstract: As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them.
By Varun Ursekar, Apaar Shanker, Yash Maurya, Shehab Yasser, Vijay S. Kalmath, Veronica Chatrath, Yuan Xue
AgentAudit is an open, extensible framework that evaluates the full lifecycle of AI agents, assessing planning, tool selection, execution, memory, and reasoning across ten dimensions such as instruction integrity, security, and alignment. Unlike existing benchmarks that focus on single aspects, AgentAudit analyzes the entire execution trace to attribute failures to specific stages. The framework was tested on five large language models, revealing significant differences in trustworthiness even among models with similar task‑completion performance.
By Shrey Nag, Sachita, Abhishek Kumar Singh, Lipi Goel, Rajeshwar Singh Janwar
arXiv:2606. 17930v1 Announce Type: new Abstract: AI evaluations are shifting toward harder tasks that benefit from longer trajectories involving tool use and iterative problem solving.
By Jessica McFadyen, Ole Jorgensen, Harry Coppock, Kevin Wei, Cozmin Ududec
HarnessFlow is a framework that automates the optimization of executable harnesses for large language model agents using a Monte Carlo Tree Search policy to balance exploration and exploitation. It refines harness candidates through a coding agent that leverages archived code, execution feedback, and reusable mechanisms, operating within a limited evaluation budget. Across six benchmarks covering agentic tasks, coding, and web interaction, HarnessFlow achieves the highest mean test scores, improving average performance by 10.89 percentage points over initial harnesses and 3.95 over Meta-Harness, while also reducing per-task execution costs.
By Zhi Hong, Qianlong Wang, Hongchen Wei, Aidan Wu, Daohong Ying, Mingze Kong, Bei Liu, Chong Luo
arXiv:2608. 09885v1 Announce Type: new Abstract: The safety of large language model (LLM) agents depends not only on model weights but also on the agent harness that manages context, memory, tools, permissions, and runtime control.
By Wanying Qu, Qinghua Mao, Yu Li, Jiyao Liu, Xin Zhang, Dadi Guo, Yanxu Zhu, Qingyu Liu, Leitao Yuan, Xi Lin, Shanfeng Zhu, Yanwei Fu, Jing Shao, Xia Hu, Dongrui Liu
arXiv:2606. 04455v1 Announce Type: new Abstract: Current AI benchmarks evaluate agents on task execution within human-designed workflows.
By Xinyu Lu, Tianshu Wang, Pengbo Wang, zujie wen, Zhiqiang Zhang, Jun Zhou, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
The study investigates whether large language models (LLMs) can autonomously build and refine a task harness—an orchestrating program that manages perception, memory, tool use, and inference—for the ARC-AGI-3 interactive benchmark. Using a fixed coding bootstrap, each model constructs a harness, operates on five development games, and revises the code twice, with each version evaluated on twenty unseen games. Results show wide variation in harness performance, modest gains from revisions, and a disconnect between development improvements and held‑out performance, indicating challenges in reliable autonomous adaptation.
By Jens Lehmann, Sahar Vahdati
arXiv:2608. 20169v1 Announce Type: cross Abstract: We present a novel approach to efficient LLM agent harness optimization through adaptive validation task selection.
By Atsuyuki Miyai, Kiyoharu Aizawa, Toshihiko Yamasaki
RankEvolve is an auto‑research framework that evolves generative ranking models by orchestrating multiple large‑language‑model coding agents through an Executable Operating Protocol (EOP). The system compiles a state machine that enforces phases, gates, branches, and loops, while a meta‑meta‑harness lets agents review and repair each other’s code. In budget‑matched experiments, heterogeneous composition of agents raised execution accuracy from 45.8 % to 62.5 % and reduced silent critical‑defect rates, achieving notable gains on the HSTU recommender and other benchmarks.
By Zheng Chen, Linfeng Liu, Hong Li, Hong Yan
arXiv:2607. 14408v1 Announce Type: new Abstract: A self-evolving agentic loop repeatedly proposes a tweaked version of an agent (its prompt template or program) and accepts or rejects the change based on a per-iteration quality signal.
By Minghao Liu, Yu Wang, Jiayun Wang, Wei Wei
The paper introduces a self‑evolving harness framework where a frozen language‑model agent first solves tasks and then edits its own harness based on run records. Using a 49‑line seed harness, the evolved harness improves average scores on in‑distribution benchmarks by 4.48 points and on out‑of‑distribution benchmarks by 12.64 points, surpassing Codex on the former and matching it on the latter. Continued evolution on a specific out‑of‑distribution benchmark further raises performance, and the study analyzes emergent mechanisms such as output truncation and history compaction.
By Qiankai Xu
arXiv:2606. 15899v1 Announce Type: cross Abstract: Open-source LLM agent ecosystems are growing rapidly, yet the security of community-contributed skills - modular tool definitions that extend agent capabilities - remains largely unvetted.
By Ismail Hossain, Sai Puppala, Md Jahangir Alam, Tanzim Ahad, Sajedul Talukder