arXiv AI

Agent-Facing Information Design in LLM Tool Registries: A Preregistered Test of Rhetoric, Position and Structure

arXiv AI
Oct 19

Advertising to Agents: Measuring Description-Framing Bias in LLM Tool Selection

The study investigates how sales language in tool registry descriptions influences AI agents’ tool selection. By creating paired listings that differ only in praise, verifiable specifications, or order, the authors found that stacked praise increased a tool’s pick rate by about 43 percentage points, while the first-listed tool was chosen roughly 72 points more often. Structured fields for numeric limits helped agents choose capable tools, but accompanying sales text could negate this benefit.

By Haochuan Kevin Wang
arXiv AI
6d ago

AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL

AgentScript‑Eval is a benchmark of 100 real tasks for Salesforce’s AgentScript DSL, testing LLMs on authoring, debugging, and optimizing conversational agents. Five models were evaluated zero‑shot, with the best achieving only 25.0% pass@3 and all scoring 0% on authoring from scratch. The study shows that model size and cost are poor predictors of success; instead, a compile‑and‑edit loop and the wording of domain guidance significantly improve performance, with the best result (37%) achieved by rewriting a skill within a live plugin.

By Dipin Khati, Shubham Mehrotra, Bin Bi, Zhou Yu, Denys Poshyvanyk, James Zhu, Sitaram Asur, Phil L. Mui
arXiv AI
Sep 3

When Agents Implement Systems: A Case Study in Defects, Detection, and Evaluation Rigor

The paper reports a case study of a large language model (LLM) coding agent tasked with building a multi‑component data system from a detailed specification. During a single session the agent introduced five defects, which were categorized by violated constraints and detection methods. The study also evaluates the agent’s retrieval‑filtering strategy on the HotpotQA benchmark, showing that filtering to a graph‑identified entity set yields higher recall than unfiltered search, with a statistically significant gap across all tested budgets.

By Phanindra Reddy Madduru
arXiv AI
Oct 26

Who Verifies the Tools? How Warped Formatters Mislead Agents and Their Verifiers

The paper investigates how tool output formatters can mislead agents and their verifiers by omitting critical details such as search limits, result caps, or identifier namespaces, even when the underlying computation is correct. Experiments with a production Retrieval-Augmented Generation (RAG) system and other open‑source agents (OpenClaw, Hermes) show that providing complete formatter information dramatically reduces unsupported agent claims and increases verifier rejection of incorrect answers. The authors provide an audit prompt that can be applied to other systems to uncover and fix similar warped formatters.

By William Jeffries
arXiv AI
Jun 12

AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility

arXiv:2606. 13608v1 Announce Type: new Abstract: Agent systems are advancing quickly across domains, but their evaluation remains fragmented.

By Xiaoyuan Liu, Jianhong Tu, Yuqi Chen, Siyuan Xie, Sihan Ren, Tianneng Shi, Gal Gantar, Evan Sandoval, Donghyun Lee, Daniel Miao, Peter J. Gilbert, Nick Hynes, Mauro Staver, Warren He, David Marn, Andrew Low, Xi Zhang, Elron Bandel, Michal Shmueli-Scheuer, Siva Reddy, Alexandre Drouin, Alexandre Lacoste, Ramayya Krishnan, Elham Tabassi, Yu Su, Victor Barres, Chenguang Wang, Wenbo Guo, Dawn Song
arXiv AI
Oct 16

Spec-Driven Development as Prescribed and as Practiced

Spec-Driven Development (SDD) is a practice where developers and AI coding agents write specifications to guide software implementation, aiming to preserve the intent behind natural‑language instructions. This study analyzes SDD through Reddit discussions, interviews with 21 practitioners (including eight tool makers), and a large corpus of 470,795 spec files from 73,030 GitHub repositories. Findings reveal that while specs often include implementation details that aid agents, they also risk decay; tool-specific structures influence perception, and benefits vary between individuals and teams, with SDD mainly used for new features and involving higher planning and review costs.

By Shyam Agarwal, Anmol Singhal, Andrew Macvean, Bogdan Vasilescu, Travis Breaux
Hugging Face Trending Papers
Jun 24

Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints

Tool Calling and Structured Output are two core capabilities of modern Agent systems, yet their interaction under joint deployment conditions remains insufficiently understood. This paper reports a reproducible phenomenon observed in a production Agent system: when Tool Calling and JSON Schema constraints are simultaneously enabled, multiple open-weight models cease invoking tools despite maintaining high schema compliance.

arXiv AI
Oct 19

Action-Space Shaping for LLM Agents: Measuring and Mitigating Tool-Schema Bias

The paper investigates how the representation of tool interfaces—referred to as a tool schema—affects the performance of large language model agents. By systematically transforming a native tool schema with nine operators, the authors demonstrate that agents exhibit significant variability in success rates across different but functionally equivalent schema variants, a phenomenon they call schema bias. Experiments on eleven LLMs, including closed models, reveal that success can range from complete failure to 97% solely due to schema differences, and that training only mitigates bias when the specific variant is included in the training data.

By Yinhong Liu, Zhili Tan, Zilin Wang, Zhijiang Guo
arXiv Computation and Language
Aug 28

Agent Seer: Synthesizing Scenarios from Specification Understanding

Agent Seer is a pipeline that automatically synthesizes realistic evaluation scenarios for AI agents that use external tools, using only the tool’s specification (function names, natural‑language descriptions, and typed parameter schemas). Starting from a single Model Context Protocol (MCP) specification, it enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock‑data‑grounded multi‑turn dialogues that demonstrate strong tool‑calling correctness and conversational coherence. Across seven diverse MCP specifications, the pipeline achieves high quality, with parameter‑schema complexity emerging as the main driver of quality variation and argument‑value accuracy identified as the dominant failure mode.

By Harish Karumuri, Mahesh Vemula, David Lopes Pegna