arXiv Machine Learning

AllocBench: Measuring Online Tool Allocation Capability in LLM Agents

arXiv:2607. 23332v2 Announce Type: replace Abstract: Creating a reusable tool is an investment: an agent pays a fixed cost now in exchange for the potential of future reuse.

arXiv Machine Learning

Jul 28

AlloBench: Measuring Online Tool Allocation Capability in LLM Agents

arXiv:2607. 23332v1 Announce Type: new Abstract: Creating a reusable tool is an investment: an agent pays a fixed cost now in exchange for the potential of future reuse.

By Daniel Wang, Andrew Xu

llms agents robotics benchmarks

arXiv AI

Aug 7

EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents

arXiv:2608. 05519v1 Announce Type: new Abstract: Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic.

By Jie Wu, Ming Gong, Feixiang Cheng, Qinqin Zhao

llms agents benchmarks

arXiv AI

Jun 30

CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents

arXiv:2511. 02734v3 Announce Type: replace Abstract: Current evaluations of Large Language Model (LLM) agents primarily emphasize task completion, often overlooking resource efficiency and adaptability.

By Jiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong, Shijue Huang, Bingxiang He, Yi R. Fung

llms agents benchmarks

arXiv AI

Jul 17

OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

arXiv:2607. 14989v1 Announce Type: cross Abstract: Large language models are increasingly evolving from text generators into general agents capable of understanding user requests, invoking external tools, and completing complex tasks through interaction.

By Chengyu Shen, Yujie Fu, Gangtao Xin, Yanheng Hou, Wenlong Fei, Guojie Zhu, Jiawei Li, Hongcheng Gao, Runming He, Zhen Hao Wong, Meiyi Qiang, Hao Liang, Zhao Cao, Hao Jiang, Chong Chen, Wentao Zhang

llms agents benchmarks

arXiv AI

Jul 7

The Remarkable Effectiveness of Providing AI Agents with Natural Language Tools: A Replication Study Validating NLT Performance Across 14 Models

arXiv:2607. 03953v1 Announce Type: cross Abstract: This study independently replicates and extends the Natural Language Tools (NLT) framework of Johnson et al.

By Alexander Somma, Isabelle Plante, Fred Premji

llms agents

arXiv AI

Jul 9

Terminus-4B: Can a Smaller Model Replace Frontier LLMs at Agentic Execution Tasks?

arXiv:2605. 03195v2 Announce Type: replace Abstract: Modern coding agents increasingly delegate specialized subtasks to subagents, which are smaller, focused agentic loops that handle narrow responsibilities like search, debugging or terminal execution.

By Spandan Garg, Vikram Nitin, Yufan Huang

llms agents reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jun 30

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.

By Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He

llms agents benchmarks

arXiv AI

Jul 15

Do AI Agents Know When a Task Is Simple? Toward Complexity-Aware Reasoning and Execution

arXiv:2607. 13034v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly automate multi-step engineering and informatics workflows, yet they rarely ask how much effort a task actually requires.

By Junjie Yin, Xinyu Feng

llms agents benchmarks

arXiv AI

Jul 7

AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments

arXiv:2607. 05174v1 Announce Type: new Abstract: Language agents, i.

By Zhiheng Xi, Dingwen Yang, Jiaqi Liu, Jixuan Huang, Honglin Guo, Baodai Huang, Tinggang Chen, Qi Zhang, Zhonghang Lu, Chenyu Liu, Jiajun Sun, Jiazheng Zhang, Dingwei Zhu, Xin Guo, Junzhe Wang, Zhihao Zhang, Yuming Yang, Junjie Ye, Minghe Gao, Dongrui Liu, Jiaming Ji, Guohao Li, Tao Gui, Qi Zhang, Xuanjing Huang

llms agents benchmarks

arXiv AI

Jun 16

ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents

arXiv:2606. 15508v1 Announce Type: new Abstract: Tool-augmented large language model agents increasingly operate over large tool libraries, but existing evaluations often focus on whether a model can call a tool correctly rather than how the visible tool menu shapes reliability, efficiency, and safety-relevant risk exposure.

By Rahul Suresh Babu, Laxmipriya Ganesh Iyer

llms agents benchmarks safety

arXiv Machine Learning

Jul 14

ToolAtlas: Learning Once, Reusing Everywhere with Tool-Side Memory

arXiv:2607. 11126v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly rely on external tools served by shared providers and accessed by heterogeneous downstream agents.

By Yue Fang, Zhibang Yang, Fangkai Yang, Xiaoting Qin, Liqun Li, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

llms agents benchmarks

arXiv AI

Jun 3

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

arXiv:2605. 28556v2 Announce Type: replace Abstract: As agent capabilities advance, existing benchmarks, such as $\tau^2$-Bench, are becoming increasingly saturated.

By Tomer Keren, Nitay Calderon, Asaf Yehudai, Yotam Perlitz, Michal Shmueli-Scheuer, Roi Reichart

llms agents benchmarks

Related stories