arXiv Machine Learning

What Git Does Not Record: Provenance Legibility in Two AI-Assisted Research Repositories

The paper audits two single‑investigator AI‑assisted research repositories to assess how well they record provenance of AI tool use. One repository contains a prescriptive trace of rule files, with 85.6% of committed lines still visible, while the other holds a narrative trace of an agent’s scratchpad, showing only 21.8% visibility. The study finds that repository‑level indicators such as the Co‑Authored‑By trailer are inconsistent and unreliable for measuring AI involvement, and it outlines which provenance claims each trace type can support or fail to support.

arXiv AI
Sep 1

CrossAudit: A Git-Native, Cross-Vendor Audit Loop for Agentic Science

CrossAudit proposes a Git‑native, cross‑vendor audit protocol for autonomous research pipelines, ensuring each work increment is reviewed by an agent from a different vendor against a human‑written rulebook. Audit outcomes, disputes, and rulings are stored as git commits, providing a replayable, versioned supervision history. The authors implemented the protocol with GitHub Actions and Python, deployed it in a computational‑chemistry pipeline, and conducted a seeded‑defect trial that revealed differing interpretations of the same rulebook by two vendors.

By Zhaohe Dong, Yuhao Chen
arXiv AI
Oct 2

Actions with Receipts: Jointly Binding Claims, Evidence, and Execution for Replayable Tool-Agent Auditing

The paper introduces a claim‑anchored execution contract that binds a tool‑using agent’s emitted claim to its exact source span, the ordered execution prefix that produced it, and the source version and access state observed. Each receipt contains deterministic anchors, source identifiers, offsets, hashes, quotes, and a domain‑separated execution commitment, allowing a verifier to reconstruct these bindings before semantic or task labels are joined. The contract defines seven independently testable properties and demonstrates high detection rates against cross‑object attacks, with strong performance on conflict‑aware support guard evaluations.

By Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Yina Sa, Daren Zha, Jun Xiao
arXiv AI
Sep 3

ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations

The paper introduces ClaimReceipt, a specification and verifier that checks whether a claim in an agent evaluation can be recomputed from retained evidence (sufficiency) and whether the evidence covers the entire experiment set (coverage). Using the CR‑2 verifier on 1,392 historical records, the authors demonstrate accurate reproduction of audit verdicts, non‑redundant field groups, and zero false positives on semantic faults. In a prospective CR‑3 run, the system correctly flags missing receipts and preserves coverage when private evidence is withheld, while adding minimal overhead to inference time and transaction size.

By Peiying Zhu, Sidi Chang
arXiv AI
3d ago

Agent or Human? Detecting Commits by AI Coding Agents in the Absence of Authorship Signatures

The study examines whether commits made by AI coding agents can be distinguished from human commits when authorship signatures are removed. Using nearly 8 million commits from over a thousand repositories, the authors find that detectors can achieve a ROC AUC of up to 0.859, largely driven by message verbosity and change size rather than code content. However, precision remains low (≈0.332) and detection performance drops on held‑out agents, indicating limited reliability without explicit attribution.

By Kosei Horikawa, Hao Li, Yutaro Kashiwa, Youssef Esseddiq Ouatiti, Mohammed Sayagh, Bram Adams, Hajimu Iida, Ahmed E. Hassan
arXiv Computation and Language
Aug 31

Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction

The paper reports that a model can pass fidelity checks—verifying that extracted values match the source—without actually opening a datasheet, due to a hidden constraint that disables tool use. To address this, the authors log every tool call in an agentic benchmark and develop two instruments: a rule‑based failure‑attribution classifier and a silent‑failure detector that flags runs based solely on which tools were invoked. While the detector shows low false positives on clean extractions and recovers all planted faults, its recall against correct tool usage but incorrect answers remains unmeasured, and a partial causal chamber confirms only a subset of claims, highlighting limitations in physical verification.

By Qing Ye, Meng-Hsuan Lin
arXiv AI
23h ago

Beyond fabrication: Misattributed numbers in papers written by AI research agents

The paper examines how AI‑written research papers often contain misattributed numerical claims. By re‑analyzing 540 judgments from a benchmark of nine AI agents on fifteen research problems, the authors manually labeled 256 claims and found that most failures stem from misalignment between the paper and the underlying work rather than outright cheating. They propose a deterministic claim‑to‑evidence checker that achieves over 90% precision and, when combined with an LLM verifier, offers a more reliable audit of AI‑generated research.

By Mosab Alfaqeeh, Omar Hourani, Muath Hourani, Hashem Alfaqeeh
Hugging Face Trending Papers
Jun 23

Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories

Generative AI coding agents are entering the open-source supply chain, yet their diverse and often invisible traces leave their prevalence poorly understood. We introduce a multi-layered detection framework that integrates configuration-file scanning, commit-message analysis, author-identity matching, and bot-signature lookup across World of Code (180M+ Git repositories), classifying agent traces into four behavioral types.