The paper introduces a six‑stage audit framework for assessing reproducibility in computer science literature and applies it to the neuro‑symbolic AI (NSAI) subfield. Using the framework, the authors screened 5,497 records, identified 1,304 eligible studies, and found verifiable code artifacts for only 455 of them. Of those, they fully or partially reproduced 85 studies, representing 6.52% of the eligible corpus and 18.68% of attempted reruns, highlighting a significant reproducibility gap even when code is declared available.
arXiv:2608. 05179v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly used across the scientific research lifecycle: ideation, literature search, experiment design and execution, analysis, manuscript drafting, and review.
arXiv:2606. 31478v1 Announce Type: new Abstract: Autonomous research agents can now draft hypotheses, write code, run experiments, and produce papers, but they remain brittle when experiments fail.
arXiv:2609.06192v1 Announce Type: new Abstract: Scientific coding agents produce interdependent code, results, figures, and claims, yet evaluating final outputs alone does not establish whether the...
arXiv:2607. 24032v1 Announce Type: new Abstract: Generative-AI evaluations can become historical before publication, yet calendar age does not affect every conclusion equally.
Generative-AI evaluations can become historical before publication, yet calendar age does not affect every conclusion equally. This paper has two linked purposes.
The paper "Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research" argues that large language model agents must faithfully implement reference methods, design experiments that truly test claims, and provide supporting evidence. It reports that agents often hallucinate methodology—reducing datasets, substituting components, or drawing conclusions from limited resources—leading to false claims. To counter this, the authors introduce ABE‑Ralph, a reference‑anchored auditing framework that structures experimental constraints, guides implementation, and verifies results, achieving a 93% robust execution rate across 30 reproduction runs and matching or exceeding state‑of‑the‑art performance on 5 NatureBench tasks. "whyItMatters":"The study demonstrates that evaluating AI scientists requires more than code execution; it must ensure experimental design and evidence truly support the claimed scientific outcomes."
arXiv:2609.08789v1 Announce Type: cross Abstract: Frontier AI developers publish safety frameworks that commit them to evidencing whether their models are dangerous. The European Union and California...
TruthInsightBench is a new benchmark designed to evaluate automated scientific discovery agents by presenting them with 40 blind tasks drawn from peer‑reviewed studies across ten domains. Each task provides only a neutral objective and frozen data, withholding source conclusions, expected values, and analysis paths, forcing agents to determine which claim the data support. A fixed LLM‑based judge scores agents on evidentiary maturity across six dimensions, using 29 artifact‑grounded items, enabling fully automated, repeatable evaluation without human grading.
arXiv:2609.05677v1 Announce Type: cross Abstract: Lifelong LLM agents increasingly rely on external skill artifacts as one element for preserving and reusing capabilities over time. These skills (usu...
arXiv:2607. 10712v1 Announce Type: cross Abstract: Scientific fraud is the instrument of doubt that malicious entities can use to establish controversy in science.
arXiv:2606. 09500v1 Announce Type: new Abstract: Objective.
arXiv:2607. 24563v1 Announce Type: new Abstract: Security Operations Centers increasingly rely on automated mapping of Cyber Threat Intelligence reports to MITRE ATT&CK, yet extractor outputs remain fallible and are often stored without the evidence, provenance, and validation history needed to decide whether an individual mapping should be trusted.