arXiv AI

Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving

arXiv:2603. 18897v2 Announce Type: replace-cross Abstract: LLM-powered agents execute tasks through a sequential loop of model generation and tool execution.

arXiv Computation and Language

3d ago

TomasuLLM: Out-of-Order Speculative Execution for LLM Agents

arXiv:2609.38201v1 Announce Type: new Abstract: Long-running tools can dominate coding-agent latency: compilers, test suites, and repository commands take seconds to minutes while the agent idles. Th...

By Jiangnan Yu, Ceyu Xu, Mengming Li, Shiyu Huang, Yiran Xia, Jian Weng, Hui Xue, Haohui Mai, Yuan Xie

llms agents benchmarks

arXiv Machine Learning

Aug 4

Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale

arXiv:2608. 00101v1 Announce Type: cross Abstract: AI coding agents like GitHub Copilot, Claude Code, and Codex interleave multi-step LLM inference with tool execution, creating a workload different from chatbots.

By Banruo Liu, Haoran Qiu, \'I\~nigo Goiri, Rodrigo Fonseca, Ricardo Bianchini, Esha Choukse

llms agents efficiency

arXiv AI

Sep 15

MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use

arXiv:2512.24565v4 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol...

By Zixiang Liu, Wenrui Liu, Elsie Dai, Wenhan Yu, Lei Yu, Tong Yang, Jinjun Han, Hong Gao

llms agents benchmarks

arXiv AI

Jul 28

SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving

arXiv:2607. 23933v1 Announce Type: cross Abstract: As LLM agents increasingly rely on the Model Context Protocol (MCP) to invoke isolated external sandboxes, disaggregated sandbox deployment introduces a fundamental tension between resource utilization and interactive tail latency.

By Yihui Zhang (Beihang University), Tianyu Wo (Beihang University), Jinghao Wang (Beihang University), Xiaoyang Sun (University of Leeds), Menghao Zhang (Beihang University), Cangzhou Yuan (Beihang University), Li Li (Beihang University), Chunming Hu (Beihang University), Albert Y. Zomaya (The University of Sydney), Renyu Yang (Beihang University)

llms rag agents

arXiv AI

Sep 1

AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios

arXiv:2605.27995v3 Announce Type: replace Abstract: Large language model (LLM)-based agents have shown strong capabilities in using external tools to solve complex tasks. However, existing evaluation...

By Kou Shi, Ziao Zhang, Shiting Huang, Avery Nie, Zhen Fang, Qiuchen Wang, Lin Chen, Huaian Chen, Zehui Chen, Feng Zhao

llms agents benchmarks

arXiv AI

4d ago

Evaluating Test-Time Scaling of General LLM Agents

arXiv:2602.18998v2 Announce Type: replace Abstract: LLM agents are increasingly expected to operate as general-purpose systems that resolve real-world user requests, yet their dynamic scaling behavio...

By Xiaochuan Li, Ryan Ming, Pranav Setlur, Abhijay Paladugu, Andy Tang, Hao Kang, Shuai Shao, Rong Jin, Chenyan Xiong

llms agents benchmarks

arXiv AI

Aug 18

From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems

arXiv:2608. 15127v1 Announce Type: cross Abstract: Agentic applications are shifting AI serving from isolated model inference to long-running workloads in which LLMs coordinate tools, environments, and persistent state.

By Chaokun Chang, Yukun Zhou, Kaihua Fu, Dakai An, Tianyu Feng, Hanfeng Lu, Sheng Yao, Pu Guo, Yinghao Yu, Yizhou Shan, Bo Li, Binhang Yuan, Wei Wang

llms agents benchmarks

arXiv AI

Jun 30

TraceLab: Characterizing Coding Agent Workloads for LLM Serving

arXiv:2606. 30560v1 Announce Type: cross Abstract: Coding agents are rapidly becoming a major application of agentic LLMs, but serving them efficiently remains challenging.

By Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy, Baris Kasikci

llms agents benchmarks

arXiv Machine Learning

Jul 10

Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems

arXiv:2607. 08010v1 Announce Type: cross Abstract: Production LLM agents often waste latency and reliability by regenerating code for the same procedural steps on every request.

By Kalle Kujanp\"a\"a, Ning Liu, Shahnawaz Alam, Yeshwanth Reddy Sura, Tianyu Yang, Kristina Klinkner, Shervin Malmasi

llms agents benchmarks

arXiv Machine Learning

Jul 14

ToolAtlas: Learning Once, Reusing Everywhere with Tool-Side Memory

arXiv:2607. 11126v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly rely on external tools served by shared providers and accessed by heterogeneous downstream agents.

By Yue Fang, Zhibang Yang, Fangkai Yang, Xiaoting Qin, Liqun Li, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

llms agents benchmarks

arXiv AI

Aug 26

PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents

PeakBench is a new benchmark designed to evaluate how large language model agents invoke multiple tools while respecting resource constraints and parallel execution. It provides executable multi‑tool workflows with dependency annotations and measured resource profiles, and introduces a two‑part evaluation framework that separates logical planning from physical scheduling. The study shows that strong logical planning alone does not guarantee safe or efficient execution, and that providing resource information can reduce overflows and improve utilization.

By Zhi-Kai Chen, Xu-Xiang Zhong, Song-Yan Li, De-Chuan Zhan, Han-Jia Ye

llms agents benchmarks

arXiv AI

Jul 17

MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers

arXiv:2607. 14642v1 Announce Type: new Abstract: As Model Context Protocol (MCP) servers emerge as the core infrastructure for connecting LLMs with external tools, existing benchmarks leverage real-world MCP servers to evaluate LLM agents' tool-using capabilities.

By Huanxi Liu, Kun Hu, Jiaqi Liao, Qiang Wang, Pengfei Qian, YuanZhao Zhai, Dawei Feng, Bo Ding, Huaimin Wang

llms agents benchmarks

Related stories