RATIO (Retrieval Across Typed Ideation Operations) is a large-scale benchmark designed to evaluate how well retrieval systems can support scientific inspiration. It defines relevance through three ideation moves—Address, Broaden, and Specify—each targeting different levels of abstraction in literature retrieval. The benchmark is built from millions of full-text CS papers using a novel discourse-marker distant supervision method, and includes extensive LLM and human vetting to ensure quality.
By Maayan Sharon, Tom Hope
arXiv:2609.24246v1 Announce Type: new
Abstract: Large language models (LLMs) have shown remarkable progress in natural language understanding, yet their effectiveness in specialized fields like astro...
By Akhil Sharma, Jatin Gupta, Ali Imam Abidi
The paper investigates whether domain-specific fine‑tuning benefits open‑ended scientific reasoning in astronomy. Using a curated 300‑question QA benchmark from 2017–2026 Olympiad‑style materials, the authors compare open‑weight, API‑served general‑purpose, multimodal, and astronomy‑specialized language models. Results show that strong general‑purpose models set the highest correctness baseline, but variations in metric agreement, judge sensitivity, benchmark composition, and modality suggest that domain specialization is task‑ and deployment‑dependent and that domain‑specific evaluation is crucial for scientific workflows.
By Vanessa Lama, Sanjay Das, Emily Herron, Yuan-Sen Ting, Tijmen de Haan, Junqi Yin, Tirthankar Ghosal, Feiyi Wang
Astro2Method is a benchmark comprising 500 expert-reviewed binary questions drawn from 346 astronomy papers, designed to evaluate large language models (LLMs) on selecting machine learning methods for astronomical problems. The study tests five LLMs, finding high accuracy (87.50‑92.55%) on Easy questions but lower performance (58.87‑64.52%) on Hard ones. Various retrieval-augmented and agent-based approaches, including web search and masked pipeline descriptions, show limited or inconsistent improvements, indicating that LLMs still struggle to infer suitable ML methods from related studies and domain knowledge.
By Xingzhong Fan, Hangfeng He, Fatemeh Nargesian
The paper introduces the Scientific Contribution Graph, a large-scale resource that extracts 6 million scientific contributions from 655 k open-access papers across multiple disciplines and links them with 36 million prerequisite edges. It frames automated technological roadmapping as the task of identifying contributions and their prerequisites, and presents a new scientific prerequisite prediction task where models forecast which existing technologies enable future discoveries. The authors report that current models achieve a 0.48 MAP score on temporally-filtered backtesting, indicating rapid progress in this area.
By Peter A. Jansen
SciRerankBench is a benchmark designed to evaluate rerankers in retrieval‑augmented large language models (RAG‑LLMs) for scientific literature question answering. It covers five scientific subjects and introduces three types of question‑context‑answer pairs—Noisy Contexts, Semantically Similar but Logically Irrelevant Contexts, and Counterfactual Contexts—to test noise resilience, relevance disambiguation, and factual consistency. The study evaluates 13 popular rerankers across five LLM families, offering detailed insights into their strengths and limitations.
By Haotian Chen, Qingqing Long, Meng Xiao, Xiao Luo, Wei Ju, Chengrui Wang, Yuanchun Zhou, Xuezhi Wang, Hengshu Zhu