arXiv Computation and Language August 28, 2026 By Maayan Sharon, Tom Hope

RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature

Read the original on arXiv Computation and Language →

RATIO (Retrieval Across Typed Ideation Operations) is a large-scale benchmark designed to evaluate how well retrieval systems can support scientific inspiration. It defines relevance through three ideation moves—Address, Broaden, and Specify—each targeting different levels of abstraction in literature retrieval. The benchmark is built from millions of full-text CS papers using a novel discourse-marker distant supervision method, and includes extensive LLM and human vetting to ensure quality.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Oct 2

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

ScholarCatalyst is a new benchmark that evaluates how well AI systems can retrieve research papers that inspire new work. The dataset was created by having 184 lead authors of 207 recent computer science papers annotate which earlier papers helped their projects, providing detailed rationales. The benchmark tests retrieval from the literature available at the start of a project, revealing that current agentic search and even advanced models like Claude Fable 5.1 perform only modestly better than simple embedding retrieval.

By Sohyeon Kim, Yoonho Lee, Bo Liu, Dayoon Ko, Rulin Shao, Seungone Kim, Graham Neubig, Pang Wei Koh, Aakanksha Chowdhery, Akari Asai, Omar Khattab, Yejin Choi, Gunhee Kim, Chelsea Finn

llms rag agents benchmarks

Hugging Face Trending Papers

Oct 6

IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas

Scientific research often begins by synthesizing ideas from a set of related papers to identify gaps and formulate new directions. However, training language models to perform this form of literature-...

llms reinforcement-learning efficiency

arXiv AI

Oct 7

IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas

arXiv:2610.08781v1 Announce Type: cross Abstract: Scientific research often begins by synthesizing ideas from a set of related papers to identify gaps and formulate new directions. However, training...

By Ziyu Chen, Yilun Zhao, Jiashuo Sun, Yiling Ma, Manasi Patwardhan, Arman Cohan

llms reinforcement-learning efficiency

arXiv AI

Aug 5

SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG

arXiv:2608. 03860v1 Announce Type: cross Abstract: We introduce SciRet, a compute-aware empirical study of retrieval-augmented generation for scientific question answering over CORD-19.

By Kaysarul Anas Apurba, Md. Hasibul Hasan, Rofiqul Alam Shehab, Asab Azad

rag nlp benchmarks

arXiv AI

Oct 15

SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generation with LLMs

SciRerankBench is a benchmark designed to evaluate rerankers in retrieval‑augmented large language models (RAG‑LLMs) for scientific literature question answering. It covers five scientific subjects and introduces three types of question‑context‑answer pairs—Noisy Contexts, Semantically Similar but Logically Irrelevant Contexts, and Counterfactual Contexts—to test noise resilience, relevance disambiguation, and factual consistency. The study evaluates 13 popular rerankers across five LLM families, offering detailed insights into their strengths and limitations.

By Haotian Chen, Qingqing Long, Meng Xiao, Xiao Luo, Wei Ju, Chengrui Wang, Yuanchun Zhou, Xuezhi Wang, Hengshu Zhu

llms rag nlp benchmarks

arXiv Computation and Language

Oct 8

Document Optimization for Black-Box Retrieval via Reinforcement Learning

arXiv:2604.05087v4 Announce Type: replace Abstract: Generative large language models (LLMs) are increasingly used as inference-time components in retrieval pipelines, for tasks such as query rewritin...

By Omri Uzan, Ron Polonsky, Douwe Kiela, Christopher Potts

llms rag reinforcement-learning multimodal

One email a morning, machine-written

Related stories

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas

IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas

SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG

SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generation with LLMs

Document Optimization for Black-Box Retrieval via Reinforcement Learning