arXiv AI

How Many Tools Should an LLM Agent See? A Chance-Corrected Answer

arXiv:2605. 24660v2 Announce Type: replace-cross Abstract: Before an LLM agent can use a tool, a retrieval system must decide which candidate tools to show to the agent.

arXiv Machine Learning
Jul 30

Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agents

arXiv:2607. 27083v1 Announce Type: new Abstract: As LLM agents increasingly depend on diverse external services such as search engines, databases, and connectors, agent harnesses face a fundamental tool-selection challenge: acquiring too few tools leaves the task under-informed, while too many adds cost, context load, and privacy exposure.

By Yicheng Feng, Yan Zhang, Yan Cheng, Wei Qi
arXiv Machine Learning
Sep 22

Toollery: Scaling LLM Agents to Thousands of Skills and Tools

Toollery is a training‑free framework that compresses candidate lists for large language model agents, enabling efficient selection from thousands of skills and tools. It generates user‑intent queries from each skill or tool specification, builds a retrieval index, and limits online selection to a compact top‑k set before the LLM makes its final decision. Evaluations on the SkillRouter benchmark, BFCL‑V4, and a proprietary smart‑cockpit dataset show that Toollery improves recall and end‑to‑end selection while keeping selection costs bounded.

By Xiangxi Tian, Ran Guan
arXiv Computation and Language
Aug 28

Agents Don't Paginate: First-Chunk Selection for LLM Tool Responses

The paper investigates why large‑language‑model coding agents rarely request a second chunk of tool output, focusing on the precision‑at‑1 rate ($p_1$) of the gold item appearing first in the first chunk. In a benchmark of 500 software‑engineering tasks, the authors compare six value functions and find that increasing $p_1$ does not systematically improve downstream accuracy; the agent can recover the correct answer from any position within the chunk. Adding file‑metadata signals to a keyword scorer actually reduces $p_1$, while a parameter‑free keyword scorer improves $p_1$ but still fails to boost overall accuracy.

By Tatiana Petrova, Andrei Mazniak, Radu State
arXiv AI
Aug 25

Enrich-Retrieve-Rank: Scaling Capability Discovery Beyond In-Context Routing

The paper introduces Enrich‑Retrieve‑Rank, a scalable method for discovering capabilities in large agent ecosystems. It replaces in‑context routing with an offline enrichment step that converts sparse metadata into searchable profiles, followed by an online retrieve‑then‑rank pipeline that returns a ranked shortlist without invoking candidates. Experiments show that as the number of capabilities grows from 10 to 7,278, the new approach maintains higher top‑1 accuracy and reduces cost by 70× compared to full‑context baselines.

By Nazib Sorathiya, Daniel Zhang, Bardiya Akhbari