arXiv Computation and Language

Checkpoints Are Not Enough: Trust Calibration in CoSLR, a Human-AI System for Systematic Literature Reviews

The paper introduces CoSLR, a Human‑AI collaborative system for systematic literature reviews that incorporates mandatory human checkpoints within a three‑phase pipeline using large language models and Retrieval‑Augmented Generation. In a survey of 63 participants, 42.9 % rated the system’s usability highly, yet 34.9 % indicated they would trust AI‑generated summaries without further human verification after brief interaction. The study highlights that effective human oversight in AI‑assisted literature reviews depends on users’ willingness to engage with the checkpoints, underscoring a calibration issue that interface design must directly address.

arXiv AI
Jul 21

AI for Auto-Research: Roadmap & User Guide

arXiv:2605. 18661v2 Announce Type: replace Abstract: AI-assisted research is crossing a threshold: fully automated systems can now generate research papers for as little as $15, while long-horizon agents can execute experiments, draft manuscripts, and simulate critique with minimal human input.

By Lingdong Kong, Xian Sun, Wei Chow, Linfeng Li, Kevin Qinghong Lin, Xuan Billy Zhang, Song Wang, Rong Li, Qing Wu, Wei Gao, Yingshuo Wang, Shaoyuan Xie, Jiachen Liu, Leigang Qu, Shijie Li, Lai Xing Ng, Benoit R. Cottereau, Ziwei Liu, Tat-Seng Chua, Wei Tsang Ooi
arXiv AI
Jun 30

meta-pipe: An LLM-agent pipeline for end-to-end automated systematic review and meta-analysis

arXiv:2606. 28363v1 Announce Type: cross Abstract: Objective: To describe the architecture and design rationale of meta-pipe, an open-source large language model (LLM)-agent pipeline that integrates the complete systematic review and meta-analysis (SR/MA) workflow -- from literature search through statistical analysis, manuscript generation, and quality assurance -- with mandatory human oversight at critical decision points.

By Hsieh-Ting Lin, Jiunn-Tyng Yeh
arXiv AI
6d ago

Trust, but verify, audit and attribute: a human-in-the-loop framework for checking AI-produced research results

The paper proposes a human‑in‑the‑loop framework for verifying AI‑generated research results, particularly in mathematics and statistics. It outlines a system where AI agents perform checks using specialized software, a fidelity audit ensures each check addresses the claim, and humans intervene at fixed points to decide on outcomes. The framework also records contributions and attributes work to both human and AI, and the authors demonstrate two automatic implementations for mathematics and statistics, including a case study where an ambiguous claim was split into two readings, one refuted and one proved.

By Gy\"orgy Eigner, Patrik P. S\"uli, Roland Holl\'os
arXiv Computation and Language
Sep 16

PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress

PaperDoctor is an agent framework that provides evidence‑grounded, actionable feedback for scientific papers before submission. It evaluates writing, layout, references, code, theory, prior work, and experiments through a three‑layer hierarchical system, linking each critique to specific evidence and revision suggestions. The system selectively rebuilds and reruns experiments to uncover reproducibility gaps, and an interactive interface lets authors explore findings tied to their manuscript.

By Kevin Qinghong Lin, Siyuan Hu, Pan Lu, Yu Chen, Yanzhe Chen, Owen Queen, Yupeng Chen, Jialin Yu, Junchi Yu, Zifeng Ding, Yuanfeng Ji, Sheng Liu, Jindong Gu, Linjie Li, Mike Zheng Shou, Philip Torr, James Zou
arXiv AI
Oct 31

Letting AI Formulate the Research Question: Autonomous Research Agents

The paper reports on an autonomous research system that allows an AI agent to formulate its own research question, screen for failure modes, and obtain critiques from a separate model before writing a paper. Over 13 development runs, minor changes in the initial instruction document caused significant shifts in the topics explored, ultimately leading to a new technology applied to a societal problem. Five fully autonomous runs produced reproducible papers that were reviewed by external AI reviewers at workshop level, with all data and code released for replication.

By Ko Fujimura
Hugging Face Trending Papers
Nov 9

Trust, but verify, audit and attribute: a human-in-the-loop framework for checking AI-produced research results

The paper proposes a human‑in‑the‑loop framework for verifying AI‑generated research results, addressing the growing volume of AI‑produced mathematics manuscripts. It outlines a system where AI agents perform checks using specialised software, but a human researcher remains involved to audit fidelity, decide at fixed points, and attribute contributions. The authors present two automatic implementations—one for mathematics (using Wolfram Language, MATLAB, Lean) and one for statistics (Python and R)—and demonstrate the mathematical tool on a two‑claim example that required splitting an ambiguous claim into a refuted and a proved reading.