arXiv:2608. 06301v1 Announce Type: new Abstract: As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them.
By Varun Ursekar, Apaar Shanker, Yash Maurya, Shehab Yasser, Vijay S. Kalmath, Veronica Chatrath, Yuan Xue
AgentAudit is an open, extensible framework that evaluates the full lifecycle of AI agents, assessing planning, tool selection, execution, memory, and reasoning across ten dimensions such as instruction integrity, security, and alignment. Unlike existing benchmarks that focus on single aspects, AgentAudit analyzes the entire execution trace to attribute failures to specific stages. The framework was tested on five large language models, revealing significant differences in trustworthiness even among models with similar task‑completion performance.
By Shrey Nag, Sachita, Abhishek Kumar Singh, Lipi Goel, Rajeshwar Singh Janwar
arXiv:2606. 17930v1 Announce Type: new Abstract: AI evaluations are shifting toward harder tasks that benefit from longer trajectories involving tool use and iterative problem solving.
By Jessica McFadyen, Ole Jorgensen, Harry Coppock, Kevin Wei, Cozmin Ududec
HarnessFlow is a framework that automates the optimization of executable harnesses for large language model agents using a Monte Carlo Tree Search policy to balance exploration and exploitation. It refines harness candidates through a coding agent that leverages archived code, execution feedback, and reusable mechanisms, operating within a limited evaluation budget. Across six benchmarks covering agentic tasks, coding, and web interaction, HarnessFlow achieves the highest mean test scores, improving average performance by 10.89 percentage points over initial harnesses and 3.95 over Meta-Harness, while also reducing per-task execution costs.
By Zhi Hong, Qianlong Wang, Hongchen Wei, Aidan Wu, Daohong Ying, Mingze Kong, Bei Liu, Chong Luo
arXiv:2608. 09885v1 Announce Type: new Abstract: The safety of large language model (LLM) agents depends not only on model weights but also on the agent harness that manages context, memory, tools, permissions, and runtime control.
By Wanying Qu, Qinghua Mao, Yu Li, Jiyao Liu, Xin Zhang, Dadi Guo, Yanxu Zhu, Qingyu Liu, Leitao Yuan, Xi Lin, Shanfeng Zhu, Yanwei Fu, Jing Shao, Xia Hu, Dongrui Liu
arXiv:2606. 04455v1 Announce Type: new Abstract: Current AI benchmarks evaluate agents on task execution within human-designed workflows.
By Xinyu Lu, Tianshu Wang, Pengbo Wang, zujie wen, Zhiqiang Zhang, Jun Zhou, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun