The paper introduces PhysClosureBench, a benchmark of 50 expert‑verified physics research tasks that test whether AI agents can organize investigations and produce evidence‑supported conclusions. Each task specifies a research goal and constraints, while the agent must design the investigation and validate evidence. Seven leading LLMs were evaluated, with Claude‑Opus‑5 scoring highest on local scientific accomplishment but achieving full research closure (RC4) only 32.7% of the time, revealing a gap between local success and complete closure.
arXiv:2609.33357v2 Announce Type: replace Abstract: Reliable automated research requires agents to vet data, verify analyses, and generate hypotheses grounded in trustworthy evidence, potentially red...
arXiv:2609.07611v1 Announce Type: new Abstract: Scientific ideation is the capacity to formulate novel and testable hypotheses from scientific evidence, and autonomous AI scientists depend on it. Exi...
PAPERBENCHX is a new benchmark that tests whether AI can reproduce scientific experiments across 12 disciplines, including electromagnetics, photonics, chemistry, materials science, biology, and robotics. It consists of 93 expert-constructed tasks, each providing a source paper, a public specification, and a domain-native scientific environment. The benchmark evaluates AI agents by having them reconstruct, execute, and diagnose key experiments, then replay submissions from scratch to score regenerated evidence against expert-validated criteria, revealing that current AI configurations fully reproduce only 14% of tasks.
arXiv:2606. 07591v1 Announce Type: cross Abstract: AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify.
arXiv:2606. 12736v1 Announce Type: new Abstract: AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood.
arXiv:2608.31076v1 Announce Type: cross Abstract: Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experi...
arXiv:2610.19159v1 Announce Type: cross Abstract: Can AI agents carry out research on open scientific problems? We introduce Into the Unknown, a benchmark that provides an agent with a science paper...
Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-end...
arXiv:2606. 11337v1 Announce Type: new Abstract: Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions.
arXiv:2607. 05682v1 Announce Type: new Abstract: LLM systems for scientific discovery increasingly assist with ideation, literature synthesis, experiment planning, and report generation, but the first research question they propose can remain difficult to audit: it may sound plausible without exposing the mechanism, falsifier, or assumption that a scientist should inspect.
arXiv:2606. 07462v1 Announce Type: new Abstract: As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution.
FrontierPhysics is a new end‑to‑end benchmark for AI agents in physics, comprising 56 tasks created by 42 physicists across nine domains of theoretical, computational, and experimental physics. Each task requires an agent to review literature, devise a research plan, conduct experiments, and produce structured outputs and manuscripts, with evaluation based on deterministic verifiers and expert rubrics. In tests of 22 models, none exceeded 46 of 168 rollouts, revealing that agents struggle mainly with rigorous reasoning and expert intuition rather than physics knowledge or execution.