RubricAtlas is a framework that extracts type-level analytical rubrics from expert reports to evaluate deep research agents. It organizes binary checkpoints around recurring analytical questions, linking them to report passages and assessing completeness separately from argument validity. Experiments on 3,079 Chinese financial reports show that individual reports cover only about half of their type’s questions, and that using different reports for requirements changes evaluation scores and agent rankings.
By Delin Wang, Sinuo Wang, Rui Sun, Beidi Luan, Qiming Shi, Jing Li, Daxin Jiang, Zuo Bai
Deep Research Bench II is a new benchmark designed to evaluate Deep Research Agents (DRAs) by requiring them to produce research reports for 132 grounded tasks across 22 domains. Each report is assessed using 9,430 fine‑grained binary rubrics that cover information recall, analysis, and presentation, all derived from expert‑written investigative articles through a rigorous LLM‑plus‑human pipeline. Evaluation of current state‑of‑the‑art DRAs shows that even the best models satisfy fewer than 50% of these rubrics, highlighting a significant gap between automated agents and human experts.
By Ruizhe Li, Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, Zhendong Mao
arXiv:2610.18092v1 Announce Type: new
Abstract: Deep research (DR) agents address open-ended information needs through multi-step reasoning, long-horizon web search, and evidence synthesis. However,...
By Yibo Wang, Senjie Jin, Yansong Ning, Xinbing Liang, Haotian Luo, Jiazheng Zhang, Yuxuan He, Will Wu, Clarenceai
FinAutoRubric is a system that lets experts define reusable evaluation guidance for financial research agents, which the agents then use to automatically generate, review, and validate task‑specific rubrics. The approach replaces costly fixed rubrics with query‑specific ones that incorporate expert standards and can be updated as new information becomes available. On three expert‑authored finance benchmarks, the automatically generated rubrics closely match human scoring and are preferred by in‑house analysts in blind reviews.
By Hoyoung Lee, Suyeol Yun, Jack Haverty, Yunju Cho, Meesong Kim, Daekyung Park, Sumin Kim, Jihoon Kwon, Jasmine Jia Geng, Andrew Chin, Yin Luo, Edward Tong, Yu Yu, Zach Golkhou, Minkyu Kim, Igor Halperin, Young Cha, Alejandro Lopez-Lira, Chanyeol Choi, Yongjae Lee
AgentHop is a diagnostic benchmark comprising 1,011 multiple‑choice questions that test agentic multi‑hop question answering over scientific citation graphs. It provides a controlled seven‑tool sandbox with fixed token, turn, and tool‑call limits, allowing researchers to dissect model performance across four operational axes: retrieval, synthesis, tool‑call, and resource management. Analysis of 19 models shows distinct family‑specific behaviors, such as GPT models committing early, Anthropic and GLM checkpoints verifying before committing, DeepSeek and Kimi over‑searching, and Gemini‑3 Pro maintaining balance, while within‑family differences are revealed in retrieval versus synthesis strategies.
By Chanhee Park, Jeongho Yoon, Sungbin Han, Hyeonseok Moon, Heuiseok Lim
arXiv:2606. 08671v1 Announce Type: new Abstract: Agent skills extend language-model agents with task-specific procedures, scripts, and references, but the tasks and environments they target continually change.
By Zhiwei Li, Yong Hu
arXiv:2608. 14747v1 Announce Type: new Abstract: WANDR (Wide ANd Deep Research) is a benchmark of 500 realistic, challenging data-collection tasks for research agents.
By Vitaliy Polshkov, Marcin Pitera, Jeremy Yang, Kirill Priemko, Maksim Gaiduk, Aleksandr Nikolenko, Denis Bykov, Clare Southern, Denis Yarats, Jerry Ma
arXiv:2609.14412v1 Announce Type: new
Abstract: Deep research agents answer complex questions through iterative loops of searching, reading, and reasoning. Recent work on reasoning-intensive benchmar...
By Radin Hamidi Rad, Amin Bigdeli, Negar Arabzadeh, Sajad Ebrahimi, Charles L. A. Clarke, Benjamin C. M. Fung, Ebrahim Bagheri
Legal Research Bench (LRB) is a new benchmark comprising 413 open-ended U.S. legal research questions, each paired with a gold answer, supporting authorities, and a binary grading rubric. The study evaluates thirteen advanced language‑model agents using web search, case‑law search, page parsing, and retrieval tools, scoring responses only when all required criteria are met and cited authorities verify. Results show that even the best model, Claude Opus 4.8, achieves full correctness on only 42.9% of questions, with performance varying by legal area and task complexity, and no clear link between more tool calls or inference cost and higher accuracy.
By Katrina Drozdov, Oliver Chen, Langston Nashold, Rayan Krishnan
arXiv:2605.29522v2 Announce Type: replace
Abstract: As scientific literature grows rapidly and research increasingly involves AI agents, automated survey generation has become a key capability for bo...
By Ziyue Yang, Da Ma, Hanqi Li, Zijian Wang, Tiancheng Huang, Zijian Hu, Chenrun Wang, Yunzhe Zhang, Xiaobao Wu, Kai Yu, Lu Chen
AgenticFinBench is a new benchmark comprising 230 real‑world financial tasks across eight categories, two languages, and four complexity levels, designed to capture the breadth and open‑ended nature of financial AI work. It introduces a reference‑free evaluation framework that uses question‑specific rubrics and a two‑stage claim audit to detect hallucinations, aligning assessment more closely with real‑world financial practice. Experiments on 18 agent configurations show that ignoring hallucinations inflates performance, that harness differences drive evidence grounding more than model differences, and that task complexity reduces quality scores, highlighting challenges in explaining financial impacts and comparing data consistently.
By Shu Zhou, Ziyang He, Yuxin Rao, Yudong Fang, Jiali Qian, Dongyuan Chen, Juntao Zhang, Yong Hu
Optimist Deep Research is an open‑source agent that conducts literature research as an iterative, state‑centric process. It breaks a research question into a tree of subquestions, uses search intents to query multiple scholarly databases, and merges results into a provenance‑preserving document registry. The agent iteratively expands the registry via citation snowballing, evaluates evidence, and refines the research plan through a critic that can add, split, rephrase, deprioritize, or close subquestions, ultimately producing a report with every citation linked to a retrieved document.
By Gihan Panapitiya