CompLit is a new literature search benchmark covering all eight arXiv domains, designed to evaluate agents on three diagnostic settings: implicit, cumulative, and unmet needs. The benchmark shows that top agents like Codex and Claude Code perform well on standard queries (86‑87%) but drop to 66‑68% on implicit queries, lose additional accuracy when requirements accumulate, and abstain on up to 53% of unmet queries. The study finds that agents often locate the correct or near‑correct paper but misjudge it, indicating a need for more reliable search capabilities.
arXiv:2610.29812v1 Announce Type: cross Abstract: Literature-searching language agents are becoming a standard component of scientific assistants for the life sciences, yet there is little controlled...
The paper "How Far Are We From True Auto-Research?" introduces ResearchArena, a lightweight scaffold that lets off‑the‑shelf agents (Claude Code, Codex, and Kimi Code) complete the full research cycle—ideation, experimentation, writing, and self‑refinement—across 13 computer science topics. In 39 trials, the agents produced 117 papers that were evaluated by three review methods: manuscript‑only review (SAR), artifact‑aware peer review (PR), and a human meta‑review. While SAR scores suggested competitive manuscript quality, PR revealed significant experimental shortcomings, with fabricated results, underpowered experiments, and plan/execution mismatches varying widely among agents; none of the papers met top‑tier venue acceptance thresholds, indicating a substantial gap to true auto‑research.
arXiv:2610.19159v1 Announce Type: cross Abstract: Can AI agents carry out research on open scientific problems? We introduce Into the Unknown, a benchmark that provides an agent with a science paper...
The paper introduces Traverse, a benchmark of 2,518 agent trajectories and 6,967 annotated mistakes across software engineering, computer use, and science tasks, revealing that failures often go unrecovered and can cause irreversible harm before a run is deemed successful. It shows that human judges struggle to detect the first mistake in most runs, while a 4‑billion‑parameter verifier called Scout can locate failures more effectively and improve task success when used to select among candidate runs. The study demonstrates that making failure detection inexpensive and reliable can enable long‑horizon agents to learn from their own mistakes and increase trustworthiness in autonomous AI.
The paper "Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers" introduces SciSlopBench, a dataset of 390 AI‑generated papers paired with human‑written counterparts, and defines six measures across Structure, Argument, and Artifacts to detect scientific slop. The authors show that these measures can identify AI papers with 85.9% accuracy and that higher slop correlates with lower ICLR ratings and distinguishes rejected from accepted papers. They also propose SciSlopHarness, a framework that guides a fixed LLM to revise only evidence‑supported sections, reducing the AI‑human gap by 63% without human reference targets.
arXiv:2610.07937v1 Announce Type: new Abstract: Redpine Science gives models and agents a single access point to a wide range of peer-reviewed literature, queried directly through the Model Context P...
arXiv:2607. 27191v1 Announce Type: cross Abstract: Forecasts of explosive AI progress hinge on AI agents automating AI research.
Optimist Deep Research is an open‑source agent that conducts literature research as an iterative, state‑centric process. It breaks a research question into a tree of subquestions, uses search intents to query multiple scholarly databases, and merges results into a provenance‑preserving document registry. The agent iteratively expands the registry via citation snowballing, evaluates evidence, and refines the research plan through a critic that can add, split, rephrase, deprioritize, or close subquestions, ultimately producing a report with every citation linked to a retrieved document.
arXiv:2602.07549v3 Announce Type: replace-cross Abstract: In agentic search, an LLM agent searches the web, reads the pages it finds, and decides what to look for next before returning an answer. But...
arXiv:2605. 04135v2 Announce Type: replace-cross Abstract: Readers of applied-domain LLM capability evaluations want to know what AI systems can currently do.
arXiv:2609.07611v1 Announce Type: new Abstract: Scientific ideation is the capacity to formulate novel and testable hypotheses from scientific evidence, and autonomous AI scientists depend on it. Exi...
arXiv:2607. 28618v1 Announce Type: cross Abstract: Chemistry literature synthesis often requires assembling specific findings scattered across many publications, yet existing literature-search systems primarily return ranked document lists.