The paper surveys how automated research systems—used for literature synthesis, ideation, workflows, writing, peer review, and end‑to‑end research—are evaluated. It compares task construction, evidence sources, evaluators, and scoring methods across six target areas, revealing that output checks, process checks, and human studies together provide a fuller picture. The authors identify gaps in diagnostic designs and offer reporting and audit recommendations to guide future evaluation efforts.
arXiv:2610.18088v2 Announce Type: new Abstract: Autoresearch systems increasingly undertake entire research projects. In practice, scientific research requires systems to pursue multiple interdepende...
arXiv:2606. 07936v1 Announce Type: cross Abstract: Human evaluation plays a critical role in assessing the quality of generated text.
arXiv:2608.31076v1 Announce Type: cross Abstract: Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experi...
arXiv:2605. 18661v2 Announce Type: replace Abstract: AI-assisted research is crossing a threshold: fully automated systems can now generate research papers for as little as $15, while long-horizon agents can execute experiments, draft manuscripts, and simulate critique with minimal human input.
arXiv:2510. 02027v2 Announce Type: replace Abstract: Scholarly publishing requires scalable scrutiny supported by auditable evidence.
Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-end...
arXiv:2608. 12788v1 Announce Type: new Abstract: The rapid advancement of Auto-Research has surfaced a fundamental evaluation challenge: how can we measure the alignment, logical coherence, and evolutionary completeness of its research trajectory with human research behavior?
The paper presents TRAILS, a taxonomy of 38 mechanisms that can undermine the reliability of large language model (LLM) read‑and‑evaluate tasks in evidence synthesis, such as study screening and risk‑of‑bias assessment. It distinguishes failures that cause inconsistent judgments across repeated runs from those that systematically bias outcomes without obvious signs. The authors outline deployment settings, testing approaches, and mitigation strategies, and argue that repeatability or accuracy alone cannot guarantee trustworthy judgments, emphasizing the need for human evaluation of evidence grounding and process transparency.
arXiv:2609.33153v2 Announce Type: replace-cross Abstract: Reported improvements from tools and reusable skills in large language model agents refer to different comparisons. This critical narrative r...
The paper "How Far Are We From True Auto-Research?" introduces ResearchArena, a lightweight scaffold that lets off‑the‑shelf agents (Claude Code, Codex, and Kimi Code) complete the full research cycle—ideation, experimentation, writing, and self‑refinement—across 13 computer science topics. In 39 trials, the agents produced 117 papers that were evaluated by three review methods: manuscript‑only review (SAR), artifact‑aware peer review (PR), and a human meta‑review. While SAR scores suggested competitive manuscript quality, PR revealed significant experimental shortcomings, with fabricated results, underpowered experiments, and plan/execution mismatches varying widely among agents; none of the papers met top‑tier venue acceptance thresholds, indicating a substantial gap to true auto‑research.
The paper introduces Automated Research Design Tracking and Assessment (ARDTrA), a system for automatically identifying and evaluating the quality of causal research designs in social science papers. An expert‑annotated dataset covering six counterfactual design families was created, and the task was tested with a multi‑turn Retrieval‑Augmented Generation pipeline across various retrieval, language‑model, and embedding configurations. Results show that passage length largely drives performance, and that the designs most challenging for machines differ from those where human experts disagree most, revealing two independent sources of difficulty.
arXiv:2606. 09809v1 Announce Type: new Abstract: AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs.