The paper proposes a human‑in‑the‑loop framework for verifying AI‑generated research results, addressing the growing volume of AI‑produced mathematics manuscripts. It outlines a system where AI agents perform checks using specialised software, but a human researcher remains involved to audit fidelity, decide at fixed points, and attribute contributions. The authors present two automatic implementations—one for mathematics (using Wolfram Language, MATLAB, Lean) and one for statistics (Python and R)—and demonstrate the mathematical tool on a two‑claim example that required splitting an ambiguous claim into a refuted and a proved reading.
CausalSmith is a framework that automates theoretical research in causal inference by integrating a Lean proof assistant with a self‑improving agentic pipeline. It uses Causalean, a Lean library of over 7,000 machine‑checked declarations, and a pipeline that selects topics, proposes results, formalizes statements, constructs proofs, and audits them against informal claims. The system’s artifacts and source code are publicly available on GitHub.
By Jiyuan Tan, Vasilis Syrgkanis
arXiv:2607. 22511v1 Announce Type: cross Abstract: Automating theoretical research is constrained not only by the generation of candidate results, but also by their reliable evaluation.
By Jiyuan Tan, Vasilis Syrgkanis
arXiv:2608. 05179v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly used across the scientific research lifecycle: ideation, literature search, experiment design and execution, analysis, manuscript drafting, and review.
By Tianyu Ding, Aditya Nannapaneni, Bingfan Liu, Ling Zhang
The paper examines how AI‑written research papers often contain misattributed numerical claims. By re‑analyzing 540 judgments from a benchmark of nine AI agents on fifteen research problems, the authors manually labeled 256 claims and found that most failures stem from misalignment between the paper and the underlying work rather than outright cheating. They propose a deterministic claim‑to‑evidence checker that achieves over 90% precision and, when combined with an LLM verifier, offers a more reliable audit of AI‑generated research.
By Mosab Alfaqeeh, Omar Hourani, Muath Hourani, Hashem Alfaqeeh
arXiv:2606. 05400v1 Announce Type: cross Abstract: Long-horizon autoformalization of research mathematics fails not only at hard lemmas, but at scale: statements drift, dependencies tangle, context decays, and local repairs corrupt distant work.
By Yuanhe Zhang, Yuekai Sun, Taiji Suzuki, Jason D. Lee, Fanghui Liu
arXiv:2607. 14582v1 Announce Type: new Abstract: Existing LLM-based theorem provers have achieved impressive results on formal mathematics benchmarks, yet they remain confined to acting as autonomous agents that prove a stated proposition.
By Junjie Zhang, Jiayu Liu, Wenbin Liu, Zhenya Huang, Doudou Wang, Yan Jiang, Leiye Xu, Tao Xiong, Wen Huang, Qi Liu, Guoping Hu, Enhong Chen, Mengping Zhang, Xiangdong Ye
As autonomous AI agents take on every stage of scientific inquiry, research output is expanding far beyond human review capacity. Yet scientific communication still relies on natural-language prose: a...
KAPPA is an open‑source toolkit that uses language‑model agents to translate LaTeX proofs from statistical papers into Lean 4, where each numbered result is verified step by step against the original text. The system assigns one of four verdicts—verified unconditionally, verified conditionally, issue recorded, or unfinished—to every result, and it can run unattended, re‑check only changed sections, and report resource usage. In tests on four example papers and three published statistical papers, KAPPA detected the known errors and additional issues that were later confirmed by hand, while a direct language‑model approach without the kit produced fewer correct verdicts and sometimes over‑claimed proofs.
By Zihua She, Xiao Wang
arXiv:2603. 02668v2 Announce Type: replace Abstract: We present SorryDB, a dynamically-updating benchmark of open Lean tasks drawn from 78 real world formalization projects on GitHub.
By Austin Letson, Leopoldo Sarra, Auguste Poiroux, Oliver Dressler, Paul Lezeau, Dhyan Aranha, Frederick Pu, Aaron Hill, Miguel Corredera Hidalgo, Julian Berman, George Tsoukalas, Lenny Taelman
The paper introduces CoSLR, a Human‑AI collaborative system for systematic literature reviews that incorporates mandatory human checkpoints within a three‑phase pipeline using large language models and Retrieval‑Augmented Generation. In a survey of 63 participants, 42.9 % rated the system’s usability highly, yet 34.9 % indicated they would trust AI‑generated summaries without further human verification after brief interaction. The study highlights that effective human oversight in AI‑assisted literature reviews depends on users’ willingness to engage with the checkpoints, underscoring a calibration issue that interface design must directly address.
By MD Aidul Islam, Malik Abdul Sami, Muhammad Waseem, Zeeshan Rasheed, Kai-kristian Kemell, Zheying Zhang, Pekka Abrahamsson
FORALL-LEAN-AGENT is a frontend‑agnostic framework that enhances auditable reasoning in formal mathematics and software verification. It integrates isolated workspaces, Lean tools, and fresh review processes—including statement comparison, axiom audits, and independent proof checking—to bind verification evidence and reviewer decisions to the same candidate artifact, ensuring traceable acceptance. Evaluations on VeriSoftBench, PutnamBench, and the Lean Eval track show that the framework improves benchmark success rates and reduces costs while providing evidence beyond aggregate solve counts.
By Naing Oo Lwin