arXiv AI By Haochuan Kevin Wang, Zechen Zhang

Agent-Facing Information Design in LLM Tool Registries: A Preregistered Test of Rhetoric, Position and Structure

Read the original on arXiv AI →

The Flow has not summarised this story yet — read it at arXiv AI.

arXiv AI
Oct 19

Advertising to Agents: Measuring Description-Framing Bias in LLM Tool Selection

The study investigates how sales language in tool registry descriptions influences AI agents’ tool selection. By creating paired listings that differ only in praise, verifiable specifications, or order, the authors found that stacked praise increased a tool’s pick rate by about 43 percentage points, while the first-listed tool was chosen roughly 72 points more often. Structured fields for numeric limits helped agents choose capable tools, but accompanying sales text could negate this benefit.

By Haochuan Kevin Wang
arXiv AI
6d ago

AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL

AgentScript‑Eval is a benchmark of 100 real tasks for Salesforce’s AgentScript DSL, testing LLMs on authoring, debugging, and optimizing conversational agents. Five models were evaluated zero‑shot, with the best achieving only 25.0% pass@3 and all scoring 0% on authoring from scratch. The study shows that model size and cost are poor predictors of success; instead, a compile‑and‑edit loop and the wording of domain guidance significantly improve performance, with the best result (37%) achieved by rewriting a skill within a live plugin.

By Dipin Khati, Shubham Mehrotra, Bin Bi, Zhou Yu, Denys Poshyvanyk, James Zhu, Sitaram Asur, Phil L. Mui
arXiv AI
Sep 3

When Agents Implement Systems: A Case Study in Defects, Detection, and Evaluation Rigor

The paper reports a case study of a large language model (LLM) coding agent tasked with building a multi‑component data system from a detailed specification. During a single session the agent introduced five defects, which were categorized by violated constraints and detection methods. The study also evaluates the agent’s retrieval‑filtering strategy on the HotpotQA benchmark, showing that filtering to a graph‑identified entity set yields higher recall than unfiltered search, with a statistically significant gap across all tested budgets.

By Phanindra Reddy Madduru
arXiv AI
Oct 26

Who Verifies the Tools? How Warped Formatters Mislead Agents and Their Verifiers

The paper investigates how tool output formatters can mislead agents and their verifiers by omitting critical details such as search limits, result caps, or identifier namespaces, even when the underlying computation is correct. Experiments with a production Retrieval-Augmented Generation (RAG) system and other open‑source agents (OpenClaw, Hermes) show that providing complete formatter information dramatically reduces unsupported agent claims and increases verifier rejection of incorrect answers. The authors provide an audit prompt that can be applied to other systems to uncover and fix similar warped formatters.

By William Jeffries