arXiv AI

DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?

arXiv:2604. 09251v3 Announce Type: replace Abstract: Deep research agents increasingly interleave web browsing with multi-step computation, yet existing benchmarks evaluate these capabilities in isolation, creating a blind spot in assessing real-world performance.

arXiv Machine Learning

Jul 31

EMBL AI Librarian: Life-Sciences Knowledge Layer for AI Agents

arXiv:2607. 28229v1 Announce Type: cross Abstract: The web is increasingly accessed by AI agents rather than humans.

By Luigi Sigillo, Matteo Silvestri, Francesco Tabaro, Rajat Bhatnagar, Syed Irtaza Mubashar, Matt Jeffryes, Daljit Nijjer, Vittorio Perera, Ola Spjuth, Julio Saez-Rodriguez, Melissa Harrison, Fabio Petroni

llms agents nlp robotics benchmarks

arXiv AI

Jul 24

AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs

arXiv:2607. 20498v1 Announce Type: new Abstract: Large language models (LLMs) augmented with tools are emerging as autonomous agents capable of using Web engine, APIs, and code to solve complex, long-horizon tasks.

By Fanjin Zhang, Zhengyang Wang, Ruixuan Huang, Kefan Zhang, Amy Xin, Yuanchun Wang, Shu Zhao, Evgeny Kharlamov, Jie Tang, Juanzi Li

llms agents nlp benchmarks

arXiv AI

Aug 11

Towards Researcher Agents for Knowledge-Graph Question Answering

arXiv:2608. 07700v1 Announce Type: new Abstract: Translating a natural-language question into a SPARQL query that can be executed against a large knowledge graph requires resolving lexical ambiguity, grounding surface terms in the target ontology, and producing graph patterns that are both syntactically valid and semantically faithful.

By Tommaso Soru, Abdulsobur Oyewale

agents nlp benchmarks

arXiv AI

Jun 10

LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake

arXiv:2606. 10460v1 Announce Type: cross Abstract: Recent large language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved.

By Haonan Wang, Jiaxiang Liu, Yurong Liu, Austin Senna Wijaya, Tianle Zhou, Eden Wu, Yijia Chen, Wanting You, Reya Vir, Daniela Pinto, Grace Fan, Yusen Zhang, Juliana Freire, Eugene Wu

llms agents nlp benchmarks

arXiv AI

Jun 2

From Graph Retrieval to Schema Realization: Counterfactual Validation for Text-to-SPARQL over Heterogeneous Knowledge Graphs

arXiv:2508. 01815v2 Announce Type: replace-cross Abstract: Text-to-SPARQL maps natural-language questions to executable SPARQL queries over RDF knowledge graphs.

By Yang Zhao, Chengxiao Dai, Yue Xiu, Dusit Niyato

agents nlp benchmarks safety

arXiv AI

Jul 23

Do Data Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval

arXiv:2605. 28787v2 Announce Type: replace-cross Abstract: In the era of autonomous agents, machine-actionable data is critical for data-driven workflows.

By Shiyu Chen, Tarfah Alrashed, Alon Halevy, Natasha Noy

llms agents

arXiv AI

Aug 3

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

arXiv:2607. 29677v1 Announce Type: new Abstract: Enterprise workflows increasingly rely on agents for \emph{schema-guided extraction}: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata.

By Boyang Zhang, Adrian Lyjak, Eli Stewart, Zhaoqi Li, Simon Suo

llms agents benchmarks

arXiv Machine Learning

Aug 12

SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Task

arXiv:2607. 24850v2 Announce Type: replace-cross Abstract: Recent advances in large language models (LLMs) have enabled search agents to autonomously tackle complex tasks across extended search and reasoning horizons.

By Lang Mei, Xiaohan Yu, Chong Chen, Liyan Liu, Xiangnan Chen, Jinchao Ma, Chao Feng, Li Huang, Siyu Mo, Sichen Kang, Yunkun Xu, Zhihan Yang, Zhujun Xue, Jingren Zhang, Qing He, Yingdi Huang, Hao Jiang, Ziao Ma, Zewei Pan, Minhao Sun, Zhuo Tao, Jinzhao Xiao, Gangtao Xin, Huanyao Zhang, Wenjian Zhang, Jiangshan Zhang, Guojie Zhu, Fangzhou Zou, Jiaxin Mao, Wentao Zhang

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 12

DailyReport: An Open-ended Benchmark for Evaluating Search Agents on Daily Search Tasks

arXiv:2606. 12871v1 Announce Type: new Abstract: Search Agents (SAs) typically leverage large language models (LLMs) to support complex information-seeking tasks by autonomously exploring web sources and synthesizing information into comprehensive responses.

By Jingxuan Han, Wei Liu, Mingyang Zhu, Youpeng Wang, Ziwen Wang, Lin Qiu, Xuezhi Cao, Xunliang Cai, Zheren Fu, Licheng Zhang, Zhendong Mao

llms agents benchmarks safety

arXiv AI

Jun 12

ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs

arXiv:2606. 12451v1 Announce Type: new Abstract: Large language models deployed as agents over large tool catalogs face a critical tool-retrieval bottleneck.

By Ashutosh Hathidara, Sai Shruthi Sistla, Sebastian Schreiber, Sahil Bansal

llms rag agents fine-tuning benchmarks

arXiv AI

Jul 14

FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.

By Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Zhiting Hu, Eric P. Xing

llms agents benchmarks

arXiv AI

Jun 6

Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)

arXiv:2606. 05901v1 Announce Type: cross Abstract: Large language models (LLMs) have fundamentally transformed the landscape of Natural Language Processing.

By Christopher J. Wedge, Joshua Stutter, Danny Dixon, Jacek Ca{\l}a

llms rag agents nlp fine-tuning benchmarks safety

Related stories