arXiv AI
SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 30
SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents
arXiv:2603. 29139v2 Announce Type: replace Abstract: Recent advances in large language models (LLMs) have enabled agentic systems to translate natural-language intent into executable scientific visualization (SciVis) tasks.
arXiv AI
4d ago
TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents
TruthInsightBench is a new benchmark designed to evaluate automated scientific discovery agents by presenting them with 40 blind tasks drawn from peer‑reviewed studies across ten domains. Each task provides only a neutral objective and frozen data, withholding source conclusions, expected values, and analysis paths, forcing agents to determine which claim the data support. A fixed LLM‑based judge scores agents on evidentiary maturity across six dimensions, using 29 artifact‑grounded items, enabling fully automated, repeatable evaluation without human grading.
arXiv AI
Jul 17
Building Agent Harnesses for Scientific Curation from Multimodal Sources
arXiv:2606. 21005v2 Announce Type: replace Abstract: Scientific discovery workflows often depend on structured curation from the literature.
arXiv AI
Jun 11
Can AI Agents Synthesize Scientific Conclusions?
arXiv:2606. 11337v1 Announce Type: new Abstract: Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions.
arXiv AI
Aug 14
OmniScientist: An Omni-Modal Omni-Discipline AI Scientist
arXiv:2608. 13558v1 Announce Type: new Abstract: Recent advances in foundation models have enabled AI scientists to automate increasingly complete research workflows, from hypothesis generation and code execution to manuscript preparation.
arXiv Computation and Language
Aug 24
Tree-of-Concerns: Hierarchical Multi-Agent Debate for Unstated-Limitation Extraction in Scientific Critique
Tree-of-Concerns is a multi‑agent framework that uses specialized skeptic personas to conduct parallel debate trees, each focusing on a specific category of potential limitations in scientific papers. The system employs structured, evidence‑grounded argumentation and a panel review mechanism to correct drift and miscalibration, ultimately extracting unstated limitations. Experiments on the ToC‑Bench benchmark show that the approach improves precision by 79% and coverage by 11% over leading baselines, providing reviewers with specific, evidence‑based concerns for systematic evaluation.