arXiv Computation and Language

Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation

arXiv AI
Oct 31

Prefix Executability: Evaluating Tool-Using Agents Beyond Final Success

The paper introduces prefix executability, a trajectory-level evaluation protocol for tool-using agents that checks whether every prefix of a predicted tool-call sequence remains executable under tool schemas, state preconditions, and data dependencies. It defines metrics such as Prefix Executability Curves, First-Error Depth, Trajectory Executability Rate, horizon-normalized AUC-PEC, and Recovery Rate to quantify reliability. Experiments on BFCL-v3, ToolSandbox, and $ au^3$-bench show that agents can achieve high final success while having low trajectory executability, revealing failure modes that final success metrics miss, and demonstrate that explicit pre-execution plan checking can improve prefix-level reliability without sacrificing terminal success.

By Amir H. Rezaeian, Weiyi Sun, Yassine Benajiba, Dan Roth
arXiv AI
3d ago

Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss

The paper introduces isloss, a deterministic oracle that tests whether conversations are preserved accurately when translated across different LLM APIs. It maps source and translated requests and replies to a shared intermediate representation, ConvIR, and applies explicit rules to detect silent information loss such as changes to instruction roles, tool interactions, error states, or generation controls. Evaluation shows that isloss can detect a majority of planted faults with few false alarms and reveals that many translations silently alter critical conversation details, affecting model behavior.

By Xin Liu, Xu Han, Yuxin Wang, Haoran Song, Mingjian Zhao, Lei Wang
arXiv AI
Oct 28

Log-To-Leak: Prompt Injection Attacks on Tool-Using LLM Agents via the Model Context Protocol

The paper "Log-To-Leak: Prompt Injection Attacks on Tool-Using LLM Agents via the Model Context Protocol" investigates how natural-language metadata in the Model Context Protocol (MCP) can be exploited to inject malicious post‑hoc logging calls into large language model agents. The authors introduce the Log-To-Leak attack, which adds an attacker‑controlled call after a legitimate tool invocation, targeting user queries, MCP server names, server responses, and agent responses. Experiments across five MCP servers and four LLM agents show high attack success rates (up to 100%) and significant logging accuracy, while existing defenses still leave substantial residual risk.

By Yuepeng Hu, Chongyu Fan, Sirat Samyoun, Jian Du
arXiv AI
Sep 2

UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities

UniACE is a unified framework that standardizes the evaluation of large language model (LLM) agents by representing each benchmark as an instruction–tool–environment triplet and running models through a shared, task‑agnostic harness in isolated runtimes. It preserves native success criteria, offers an offline mode for dynamic‑resource tasks, and standardizes efficiency metrics, execution records, and failure attribution. Applying UniACE to 7 benchmarks across 24 domains and 15 models revealed significant score shifts, ranking reversals, and sensitivity to evidence representation, highlighting the impact of evaluation configuration on reported agent performance.

By Pengyu Zhu, Lijun Li, Yaxing Lyu, Qianxin Luo, Jingyi Yang, Yi Liu, Tingfeng Hui, Xinyu Yuan, Li Sun, Sen Su, Jing Shao
Hugging Face Trending Papers
Jun 24

Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints

Tool Calling and Structured Output are two core capabilities of modern Agent systems, yet their interaction under joint deployment conditions remains insufficiently understood. This paper reports a reproducible phenomenon observed in a production Agent system: when Tool Calling and JSON Schema constraints are simultaneously enabled, multiple open-weight models cease invoking tools despite maintaining high schema compliance.