arXiv Computation and Language

RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature

RATIO (Retrieval Across Typed Ideation Operations) is a large-scale benchmark designed to evaluate how well retrieval systems can support scientific inspiration. It defines relevance through three ideation moves—Address, Broaden, and Specify—each targeting different levels of abstraction in literature retrieval. The benchmark is built from millions of full-text CS papers using a novel discourse-marker distant supervision method, and includes extensive LLM and human vetting to ensure quality.

arXiv AI

Oct 2

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

ScholarCatalyst is a new benchmark that evaluates how well AI systems can retrieve research papers that inspire new work. The dataset was created by having 184 lead authors of 207 recent computer science papers annotate which earlier papers helped their projects, providing detailed rationales. The benchmark tests retrieval from the literature available at the start of a project, revealing that current agentic search and even advanced models like Claude Fable 5.1 perform only modestly better than simple embedding retrieval.

By Sohyeon Kim, Yoonho Lee, Bo Liu, Dayoon Ko, Rulin Shao, Seungone Kim, Graham Neubig, Pang Wei Koh, Aakanksha Chowdhery, Akari Asai, Omar Khattab, Yejin Choi, Gunhee Kim, Chelsea Finn

llms rag agents benchmarks

Hugging Face Trending Papers

Oct 6

IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas

Scientific research often begins by synthesizing ideas from a set of related papers to identify gaps and formulate new directions. However, training language models to perform this form of literature-...

llms reinforcement-learning efficiency

arXiv AI

Oct 7

IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas

arXiv:2610.08781v1 Announce Type: cross Abstract: Scientific research often begins by synthesizing ideas from a set of related papers to identify gaps and formulate new directions. However, training...

By Ziyu Chen, Yilun Zhao, Jiashuo Sun, Yiling Ma, Manasi Patwardhan, Arman Cohan

llms reinforcement-learning efficiency

arXiv AI

Aug 5

SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG

arXiv:2608. 03860v1 Announce Type: cross Abstract: We introduce SciRet, a compute-aware empirical study of retrieval-augmented generation for scientific question answering over CORD-19.

By Kaysarul Anas Apurba, Md. Hasibul Hasan, Rofiqul Alam Shehab, Asab Azad

rag nlp benchmarks

arXiv AI

Oct 15

SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generation with LLMs

SciRerankBench is a benchmark designed to evaluate rerankers in retrieval‑augmented large language models (RAG‑LLMs) for scientific literature question answering. It covers five scientific subjects and introduces three types of question‑context‑answer pairs—Noisy Contexts, Semantically Similar but Logically Irrelevant Contexts, and Counterfactual Contexts—to test noise resilience, relevance disambiguation, and factual consistency. The study evaluates 13 popular rerankers across five LLM families, offering detailed insights into their strengths and limitations.

By Haotian Chen, Qingqing Long, Meng Xiao, Xiao Luo, Wei Ju, Chengrui Wang, Yuanchun Zhou, Xuezhi Wang, Hengshu Zhu

llms rag nlp benchmarks

arXiv Computation and Language

Oct 8

Document Optimization for Black-Box Retrieval via Reinforcement Learning

arXiv:2604.05087v4 Announce Type: replace Abstract: Generative large language models (LLMs) are increasingly used as inference-time components in retrieval pipelines, for tasks such as query rewritin...

By Omri Uzan, Ron Polonsky, Douwe Kiela, Christopher Potts

llms rag reinforcement-learning multimodal

arXiv AI

Jul 28

VecTree-RAG: An Agentic Retrieval-Augmented Generation Framework Combining Vector and Tree Retrieval for Efficiency and Accuracy

arXiv:2607. 23006v1 Announce Type: cross Abstract: Scientific question answering requires a retrieval system to solve two distinct problems: identifying which papers are relevant and locating the supporting evidence within those papers.

By Xinyan Zhong, Yuwei Shi, Yuqi Wei, Chen Shen, Tianhang Zhou, Zhenghao Wu

llms rag agents nlp benchmarks

arXiv AI

Oct 16

Do Agents Need a New Kind of Paper? SPRA: Structured Paper Representation for Agents

The paper introduces SPRA (Structured Paper Representation for Agents), a comprehensive representation of AI research papers that organizes key research elements—Problems, Contributions, Components, ExperimentSetups, Measures, and Findings—into typed relations for easier querying, traversal, and reuse. SPRA includes paper-level citation relations that capture dependencies, component usage, and comparisons, and it has been instantiated across more than one million AI papers, yielding millions of research units and internal relations. Empirical evaluations show that SPRA improves source-conditioned scientific QA and directed cross-paper reasoning over passage retrieval while significantly reducing input and output token counts, demonstrating its practicality for scientific workflows.

By Yutong Leng, Haojiong Chen, Fukai Shang, Xiaoyang Wang, Mindong Liu, Runyuan Ma, Jiantao Qiu

rag agents

arXiv Computation and Language

Oct 27

A Domain-Aware Pipeline for Structured Extraction and Retrieval-Augmented Question Answering over arXiv Superconductivity Literature

arXiv:2610.34391v1 Announce Type: cross Abstract: Scientific literature in materials physics is growing faster than researchers can manually inspect it, yet most retrieval tools rely on keyword match...

By Ivan Iorsh, Aleksandr Kapitonov, Ivan Filianin, Matvey Lesonen, Nikita Chizhov

llms rag nlp benchmarks

arXiv AI

Sep 25

Learning to Ideate for Scientific Impact

The paper "Learning to Ideate for Scientific Impact" explores using delayed signals of scientific uptake—specifically citation-normalized impact—as feedback to steer large language models toward generating high‑impact research ideas. The authors build a dataset of over 100,000 computer science papers, train a reward model to predict citation impact from goal‑idea pairs, and align an idea generator via supervised fine‑tuning and reinforcement learning. Evaluation with a reference‑grounded protocol shows that the RL‑tuned model consistently produces ideas with higher estimated impact than baseline models.

By Shubham Kale, Aniketh Garikaparthi, Manasi Patwardhan

llms reinforcement-learning fine-tuning

arXiv AI

Oct 27

Navigating Ideation Space: Decomposed Conceptual Representations for Positioning Scientific Ideas

arXiv:2601.08901v2 Announce Type: replace-cross Abstract: Scientific discovery requires new ideas to be situated within an ever-expanding landscape of existing knowledge. An emerging challenge is how...

By Minqian Liu, Yuexi Shen, Dawei Zhou, Lifu Huang

llms rag

arXiv AI

Jun 19

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments

arXiv:2606. 20235v1 Announce Type: cross Abstract: Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, intent-driven literature exploration.

By Tingyue Pan, Mingyue Cheng, Daoyu Wang, Yitong Zhou, Jie Ouyang, Qi Liu, Enhong Chen

llms agents benchmarks

Related stories