arXiv AI June 12, 2026 By Ashutosh Hathidara, Sai Shruthi Sistla, Sebastian Schreiber, Sahil Bansal

ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs

arXiv:2606. 12451v1 Announce Type: new Abstract: Large language models deployed as agents over large tool catalogs face a critical tool-retrieval bottleneck.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Jun 3

Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition

arXiv:2606. 03657v1 Announce Type: new Abstract: Large language models for code generation often need to use APIs that are absent from their pretraining data.

By Jinnuo Liu, Yue Peng, Jinhan Niu, Hongyi Wen

llms agents fine-tuning benchmarks

arXiv AI

Jul 28

TRACE: Business Rule-Grounded Reasoning Curriculum for Knowledge-Preserving Parametric Tool Retrieval in Enterprise LLMs

arXiv:2607. 22639v1 Announce Type: new Abstract: Parametric retrieval enables LLMs to retrieve tools implicitly by assigning each API a unique virtual token and training the model to generate it via constrained beam search.

By Sai Shruthi Sistla, Ashutosh Hathidara, Christopher Toukmaji, Mayank Shrivastava, Karthikeyan Asokkumar

llms rag fine-tuning

arXiv AI

Jul 29

Tools Are Not Islands: Set-Level Tool Retrieval for LLM Agents via Query-Conditioned Hyperedge Prediction

arXiv:2607. 25718v1 Announce Type: cross Abstract: Large language model (LLM) agents increasingly rely on invoking external tools to complete real-world tasks.

By Xinyi Hong, Pinjun Dong, Xinyang Yu, Binyan Jiang

llms agents benchmarks

arXiv AI

Jun 4

QO-Bench: Diagnosing Query-Operator-Preserving Retrieval over Typed Event Tuples

arXiv:2606. 04646v1 Announce Type: cross Abstract: Many real-world questions over business, legal, and scientific corpora are natural-language versions of database-style queries over records latent in text.

By Mengao Zhang, Xiang Yang, Chang Liu, Tianhui Tan, Ke-wei Huang

llms rag nlp benchmarks data-engineering

arXiv AI

Jul 8

PORTS: Preference-Optimized Retrievers for Tool Selection with Large Language Models

arXiv:2607. 05441v1 Announce Type: cross Abstract: Integrating external tools with Large Language Models (LLMs) has emerged as a promising paradigm for accomplishing complex tasks.

By Lorenzo Molfetta, Giacomo Frisoni, Nicol\`o Monaldini, Gianluca Moro

llms safety

arXiv AI

6d ago

VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

arXiv:2608. 12282v1 Announce Type: new Abstract: Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation.

By Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder, Siyu Huo, Raavi Gupta, Abhinav Jain, Praveen Venkateswaran, Abdulhamid Adebayo, Danish Contractor

agents benchmarks

Related stories

Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition

TRACE: Business Rule-Grounded Reasoning Curriculum for Knowledge-Preserving Parametric Tool Retrieval in Enterprise LLMs

Tools Are Not Islands: Set-Level Tool Retrieval for LLM Agents via Query-Conditioned Hyperedge Prediction

QO-Bench: Diagnosing Query-Operator-Preserving Retrieval over Typed Event Tuples

PORTS: Preference-Optimized Retrievers for Tool Selection with Large Language Models

VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies