arXiv Computation and Language August 28, 2026 By Maayan Sharon, Tom Hope

RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature

Read the original on arXiv Computation and Language →

RATIO (Retrieval Across Typed Ideation Operations) is a large-scale benchmark designed to evaluate how well retrieval systems can support scientific inspiration. It defines relevance through three ideation moves—Address, Broaden, and Specify—each targeting different levels of abstraction in literature retrieval. The benchmark is built from millions of full-text CS papers using a novel discourse-marker distant supervision method, and includes extensive LLM and human vetting to ensure quality.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Aug 5

SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG

arXiv:2608. 03860v1 Announce Type: cross Abstract: We introduce SciRet, a compute-aware empirical study of retrieval-augmented generation for scientific question answering over CORD-19.

By Kaysarul Anas Apurba, Md. Hasibul Hasan, Rofiqul Alam Shehab, Asab Azad

rag nlp benchmarks

arXiv AI

Jul 28

VecTree-RAG: An Agentic Retrieval-Augmented Generation Framework Combining Vector and Tree Retrieval for Efficiency and Accuracy

arXiv:2607. 23006v1 Announce Type: cross Abstract: Scientific question answering requires a retrieval system to solve two distinct problems: identifying which papers are relevant and locating the supporting evidence within those papers.

By Xinyan Zhong, Yuwei Shi, Yuqi Wei, Chen Shen, Tianhang Zhou, Zhenghao Wu

llms rag agents nlp benchmarks

arXiv AI

Jun 19

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments

arXiv:2606. 20235v1 Announce Type: cross Abstract: Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, intent-driven literature exploration.

By Tingyue Pan, Mingyue Cheng, Daoyu Wang, Yitong Zhou, Jie Ouyang, Qi Liu, Enhong Chen

llms agents benchmarks

arXiv Computation and Language

3d ago

Lit3R: Retrieve-Relate-Read for Evidence-Grounded Question Answering over Scientific Literature

Lit3R is a system developed by tus-nlp for the LitTraceQA shared task, which focuses on evidence-grounded question answering over scientific literature. The system combines off-the-shelf retrieval, reranking, and large language model components without task-specific training, using an iterative retrieval process that merges BM25-based sparse and dense retrieval, cross-encoder reranking, and LLM verification, along with paper-to-paper expansion. In the official test set, Lit3R achieved a 4th place ranking on the leaderboard.

By Akira Ise, Kotaro Kumagai, Yuta Yamaguchi, Hisanori Ozaki, Yukio Uematsu, Ikuya Yamada

llms rag nlp benchmarks

arXiv AI

Jul 24

SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration

arXiv:2607. 20926v1 Announce Type: new Abstract: Scientific research involves complex information-seeking and reasoning workflows across heterogeneous sources.

By Yinhao Tang, Youqing Fang, Yanan Sun, Wenran Liu, Weiming Zhang, Bin Liu, Kuikun Liu, Wenwei Zhang, Kai Chen

llms agents nlp benchmarks

arXiv AI

Jul 9

RWGBench: Evaluating Scholarly Positioning in Related Work Generation

arXiv:2606. 24894v2 Announce Type: replace-cross Abstract: Large language models have shown strong fluency in scientific writing, yet the evaluation of related work generation (RWG) remains limited.

By Anzhe Xie, Weihang Su, Jiaxin Mao, Yiqun Liu, Shaoping Ma, Qingyao Ai

llms nlp benchmarks

One email a morning, machine-written

Related stories

SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG

VecTree-RAG: An Agentic Retrieval-Augmented Generation Framework Combining Vector and Tree Retrieval for Efficiency and Accuracy

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments

Lit3R: Retrieve-Relate-Read for Evidence-Grounded Question Answering over Scientific Literature

SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration

RWGBench: Evaluating Scholarly Positioning in Related Work Generation