arXiv Machine Learning

The Calibration Turn in AI-Assisted Research: A Conceptual and Methodological Framework for Evidence-Licensed Claims

arXiv:2606. 31273v1 Announce Type: new Abstract: AI-assisted research has entered a stage in which the central question is not only whether systems can generate hypotheses, run experiments, or produce manuscripts, but whether their scientific claims are calibrated to the evidence that supports them.

arXiv AI
Jun 18

Externalizing Research Synthesis and Validation in AI Scientists through a Research Harness

arXiv:2606. 18874v1 Announce Type: new Abstract: AI systems can increasingly automate scientific workflows, but the reasoning that links prior evidence, generated ideas, experiments and final claims often remains implicit inside model inference.

By Zijian Wang, Hanqi Li, Ziyue Yang, Zijian Hu, Shenghan Zuo, Yunzhe Zhang, Da Ma, Danyu Luo, Chenrun Wang, Jing Peng, Tiancheng Huang, Sijia Guo, Huayang Wang, Zichen Zhu, Senyu Han, Yilu Cao, Kai Yu, Lu Chen
arXiv AI
Aug 26

Ethical LLM-Assisted Research: A Framework for Responsible Delegation, Verification, and Epistemic Value

The paper proposes a normative framework for ethical use of large language models (LLMs) in scientific research, treating reasoning as a distributed process where human control remains essential for epistemic legitimacy. It introduces key constructs—content origin, human verification, responsibility assignment, accountable ownership, and epistemic outcome—to separate claim provenance from verification and responsibility. The authors argue that the ethical boundary hinges on adequate verification and accountable human ownership, and they propose an "epistemic audit" to document delegation, verification, provenance, and responsibility for transparent, reviewable AI-assisted reasoning.

By Kalin Stoyanov
arXiv AI
Sep 7

TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents

TruthInsightBench is a new benchmark designed to evaluate automated scientific discovery agents by presenting them with 40 blind tasks drawn from peer‑reviewed studies across ten domains. Each task provides only a neutral objective and frozen data, withholding source conclusions, expected values, and analysis paths, forcing agents to determine which claim the data support. A fixed LLM‑based judge scores agents on evidentiary maturity across six dimensions, using 29 artifact‑grounded items, enabling fully automated, repeatable evaluation without human grading.

By Zhibo Yang, Chen Zhang, Yuewei Zhang, Hao Wang
arXiv AI
Aug 21

Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis

arXiv:2608. 19902v1 Announce Type: new Abstract: AI agents can execute scientific analyses, but an analytic output becomes a defensible claim only after alternatives are weighed and the claim is limited to what the evidence supports.

By Zijiao Chen, Nicholas Lu, Xinhui Li, Jocelyn A. Ricard, Ce Ju, Huan H. Wang, Christian Kindermann, Jeanette A. Mumford, Steven Dillmann, James Kent, Alejandro de la Vega, Sanmi Koyejo, Vince D. Calhoun, Joshua W. Buckholtz, Juan Helen Zhou, Steffen Bollmann, Russell A. Poldrack
arXiv AI
Aug 25

CausalSmith: A Formally Grounded, Self-Improving Agentic Framework for Automated Research in Causal Inference

CausalSmith is a framework that automates theoretical research in causal inference by integrating a Lean proof assistant with a self‑improving agentic pipeline. It uses Causalean, a Lean library of over 7,000 machine‑checked declarations, and a pipeline that selects topics, proposes results, formalizes statements, constructs proofs, and audits them against informal claims. The system’s artifacts and source code are publicly available on GitHub.

By Jiyuan Tan, Vasilis Syrgkanis