arXiv AI
RubricAtlas: Mining Type-Level Rubrics from Expert Reports to Evaluate Deep Research Agents
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 11
DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
Deep Research Bench II is a new benchmark designed to evaluate Deep Research Agents (DRAs) by requiring them to produce research reports for 132 grounded tasks across 22 domains. Each report is assessed using 9,430 fine‑grained binary rubrics that cover information recall, analysis, and presentation, all derived from expert‑written investigative articles through a rigorous LLM‑plus‑human pipeline. Evaluation of current state‑of‑the‑art DRAs shows that even the best models satisfy fewer than 50% of these rubrics, highlighting a significant gap between automated agents and human experts.
arXiv AI
Oct 15
LifeSearch: Benchmarking Deep Research Agents for Real-World Life and Productivity
arXiv:2610.18092v1 Announce Type: new Abstract: Deep research (DR) agents address open-ended information needs through multi-step reasoning, long-horizon web search, and evidence synthesis. However,...
arXiv AI
2d ago
FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents
FinAutoRubric is a system that lets experts define reusable evaluation guidance for financial research agents, which the agents then use to automatically generate, review, and validate task‑specific rubrics. The approach replaces costly fixed rubrics with query‑specific ones that incorporate expert standards and can be updated as new information becomes available. On three expert‑authored finance benchmarks, the automatically generated rubrics closely match human scoring and are preferred by in‑house analysts in blind reviews.
arXiv Machine Learning
Aug 18
WANDR: A Benchmark for Wide and Deep Research
arXiv:2608. 14747v1 Announce Type: new Abstract: WANDR (Wide ANd Deep Research) is a benchmark of 500 realistic, challenging data-collection tasks for research agents.
arXiv AI
Jul 29
Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
arXiv:2607. 25891v1 Announce Type: new Abstract: Evaluating AI agents in interactive environments is hindered by fragmented tasks, scaffolds, verifiers, and scoring rules.
arXiv AI
Sep 15
Question's Gambit: The First Move Matters in Agentic Deep Search
arXiv:2609.14412v1 Announce Type: new Abstract: Deep research agents answer complex questions through iterative loops of searching, reading, and reasoning. Recent work on reasoning-intensive benchmar...