UniACE is a unified framework that standardizes the evaluation of large language model (LLM) agents by representing each benchmark as an instruction–tool–environment triplet and running models through a shared, task‑agnostic harness in isolated runtimes. It preserves native success criteria, offers an offline mode for dynamic‑resource tasks, and standardizes efficiency metrics, execution records, and failure attribution. Applying UniACE to 7 benchmarks across 24 domains and 15 models revealed significant score shifts, ranking reversals, and sensitivity to evidence representation, highlighting the impact of evaluation configuration on reported agent performance.
By Pengyu Zhu, Lijun Li, Yaxing Lyu, Qianxin Luo, Jingyi Yang, Yi Liu, Tingfeng Hui, Xinyu Yuan, Li Sun, Sen Su, Jing Shao
arXiv:2607. 23933v1 Announce Type: cross Abstract: As LLM agents increasingly rely on the Model Context Protocol (MCP) to invoke isolated external sandboxes, disaggregated sandbox deployment introduces a fundamental tension between resource utilization and interactive tail latency.
By Yihui Zhang (Beihang University), Tianyu Wo (Beihang University), Jinghao Wang (Beihang University), Xiaoyang Sun (University of Leeds), Menghao Zhang (Beihang University), Cangzhou Yuan (Beihang University), Li Li (Beihang University), Chunming Hu (Beihang University), Albert Y. Zomaya (The University of Sydney), Renyu Yang (Beihang University)
arXiv:2606. 01533v1 Announce Type: cross Abstract: Computer use agents (CUAs) today are primarily deployed as single serial agents.
By Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried
Harness-Bench is a diagnostic benchmark that evaluates how different harness configurations affect LLM agent performance in realistic workflows. It tests 106 sandboxed offline tasks across multiple model backends, recording artifacts, execution traces, and usage statistics to analyze completion, process quality, efficiency, and failure behavior. The study finds significant variation in outcomes across model-harness pairings, indicating that agent capability should be reported at the configuration level rather than solely by the base model.
By Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Zongwei Lv, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Guangxiang Zhao, Tong Yang
AgentPerfBench is a new benchmarking suite designed to evaluate the inference performance of agentic large language models (LLMs) that handle multi‑turn, tool‑using, and context‑expanding tasks. It builds on real traces from agentic benchmarks such as SWE‑Bench and TerminalBench, and generates synthetic profiles that reflect realistic input/output lengths and turn counts. The suite also provides kernel‑level Nsight Compute traces and a multi‑dimensional roofline model to identify hardware bottlenecks and quantify the gap between traditional chat benchmarks and agentic workloads.
arXiv:2606. 19382v1 Announce Type: cross Abstract: While LLM-powered agents offer end-to-end automation for industrial asset lifecycles, real-world Industry 4.
By Kanishk Kushwaha, Vikrant Vinod Bansode, Harsh Vardhan, Dhaval C. Patel