arXiv AI
Jul 3
Coding-agents can replicate scientific machine learning papers
arXiv:2607. 02134v1 Announce Type: new Abstract: Scientific machine learning papers typically make computational claims, e.
Position: Let's Strengthen Verifiability If We Can't Enforce Reproducibility
The paper argues that empirical results in Machine Learning are often difficult to reproduce due to limited availability of code and supporting materials, which hampers research progress. It analyzes and quantifies these challenges and proposes concrete measures to enhance the verifiability of results, even if full reproducibility cannot be guaranteed. The authors provide their code and supporting resources on GitHub for reference.
Related stories
arXiv Machine Learning
Sep 16
OPEN-1B: A Fully Auditable Training Run
The paper introduces Open-1B, a language model trained under a new fully auditable regime that ensures every training operation is reproducible on heterogeneous commodity hardware with bitwise certainty. By enforcing a fixed order on sources of nondeterminism—GPU reductions, data batch ordering, and inter/intra-node communication—the authors enable auditors to replay and verify individual training steps on a single machine. The release includes the full pretraining dataset, all intermediate checkpoints, the training codebase, and an audit harness for step-by-step verification.
arXiv AI
Aug 28
Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research
The paper "Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research" argues that large language model agents must faithfully implement reference methods, design experiments that truly test claims, and provide supporting evidence. It reports that agents often hallucinate methodology—reducing datasets, substituting components, or drawing conclusions from limited resources—leading to false claims. To counter this, the authors introduce ABE‑Ralph, a reference‑anchored auditing framework that structures experimental constraints, guides implementation, and verifies results, achieving a 93% robust execution rate across 30 reproduction runs and matching or exceeding state‑of‑the‑art performance on 5 NatureBench tasks. "whyItMatters":"The study demonstrates that evaluating AI scientists requires more than code execution; it must ensure experimental design and evidence truly support the claimed scientific outcomes."
arXiv AI
Aug 24
AI with Authority, from Application to Silicon
arXiv:2608.21356v1 Announce Type: cross Abstract: For sixty years, machine verification has been a major cost overhead, affordable only for exceptional artifacts. Here we report that generative AI in...
arXiv Machine Learning
Jul 8
Fine-Tuning Integrity for Modern Neural Networks: Structured Drift Proofs via Norm, Rank, and Sparsity Certificates
arXiv:2604. 04738v2 Announce Type: replace-cross Abstract: Fine-tuning is the dominant paradigm for adapting large machine learning models, yet current deployment pipelines provide no way to verify how a released model was updated.
arXiv AI
Aug 24
Prediction certification cannot replace explanation certification: a competence envelope for trustworthy AI under compound stress
The paper argues that prediction‑based certifications—such as accuracy, calibration, and conformal coverage—are insufficient to guarantee trustworthy AI. It proves a separation theorem showing that a model can appear reliable under all prediction‑side certificates yet differ arbitrarily in explanation fidelity and deployment behaviour. The authors propose a competence envelope framework that combines both prediction and explanation certification to detect such hidden failures.
arXiv Machine Learning
Jun 30
Proofs of Ownership for Machine Learning Models
arXiv:2606. 30423v1 Announce Type: new Abstract: With the increasing adoption of Machine Learning, protecting model ownership has become an essential challenge.
arXiv AI
Jun 17
ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
arXiv:2606. 18237v1 Announce Type: cross Abstract: Reproducing research results from papers and released code is central to scientific progress.
arXiv AI
Jun 16
Evaluating the Robustness of Proof Autoformalization in Lean 4
arXiv:2606. 14867v1 Announce Type: cross Abstract: Proof autoformalization aims to translate a mathematical informal proof written in natural language into a formal proof in a formal language such as Lean~4.
arXiv Machine Learning
Jul 20
Publicly-Verifiable Certificates for Statistical Algorithms
arXiv:2607. 15528v1 Announce Type: new Abstract: Following Goldwasser, Rothblum, Shafer, and Yehudayoff, who defined a framework for interactive proofs of learning [ITCS'21], we initiate the study of non-interactive proofs of learning.
arXiv Machine Learning
Jun 30
Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models
arXiv:2606. 29346v1 Announce Type: new Abstract: Post-hoc explanation methods are routinely used to interpret scientific machine learning models, with the deliverable understood to be insight into the phenomenon the model has been trained on.
arXiv AI
Jun 16
Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs
arXiv:2606. 15258v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly capable of mathematical problem solving and can even assist with research-level proofs, yet we still lack a scalable and reproducible way to measure step-level reasoning in long proofs across diverse sources.