arXiv:2602. 11354v3 Announce Type: replace Abstract: The literature has witnessed an emerging interest in AI agents for automated assessment of scientific papers.
arXiv:2409. 11363v2 Announce Type: replace-cross Abstract: AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research.
The paper introduces AutoMat, a benchmark designed to test large language model (LLM) coding agents on their ability to reproduce claims from computational materials science. AutoMat presents three challenges: reconstructing underspecified procedures, navigating specialized toolchains, and assessing whether the evidence supports a claim. Experiments show that current LLM agents achieve low success rates, with the best setting reaching only 53%, and failures stem mainly from incomplete procedures, methodological deviations, and execution fragility.
arXiv:2607. 02931v1 Announce Type: new Abstract: AI tools are accelerating scientific publication while the systems that review it struggle to keep up, and independent verification of published research has become both harder and more important.
arXiv:2608. 05179v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly used across the scientific research lifecycle: ideation, literature search, experiment design and execution, analysis, manuscript drafting, and review.
arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.
TruthInsightBench is a new benchmark designed to evaluate automated scientific discovery agents by presenting them with 40 blind tasks drawn from peer‑reviewed studies across ten domains. Each task provides only a neutral objective and frozen data, withholding source conclusions, expected values, and analysis paths, forcing agents to determine which claim the data support. A fixed LLM‑based judge scores agents on evidentiary maturity across six dimensions, using 29 artifact‑grounded items, enabling fully automated, repeatable evaluation without human grading.
arXiv:2606. 12736v1 Announce Type: new Abstract: AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood.
arXiv:2606. 07591v1 Announce Type: cross Abstract: AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify.
arXiv:2606. 30182v1 Announce Type: new Abstract: AI models are rapidly improving at autonomous coding, as shown by benchmark progress and one-off demonstrations such as AI implementing a C compiler.
arXiv:2609.24911v1 Announce Type: new Abstract: Social simulation offers the social sciences an experimental instrument that the real world cannot supply, and generative agents have transformed it by...
arXiv:2606. 18237v1 Announce Type: cross Abstract: Reproducing research results from papers and released code is central to scientific progress.
arXiv:2606. 26158v1 Announce Type: new Abstract: When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version.