arXiv Machine Learning By Yanhang Li, Zhichao Fan, Zexin Zhuang

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits

Read the original on arXiv Machine Learning →

arXiv:2607. 02586v1 Announce Type: new Abstract: Governance frameworks ask AI providers and auditors for documented evaluation evidence, and perturbation-based construct-validity audits are a common form of that evidence.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Jun 2

Make Mechanistic Interpretability Auditable: A Call to Develop Guidelines via Continuous Collaborative Reviewing

arXiv:2606. 00033v1 Announce Type: cross Abstract: While mechanistic interpretability (MI) has produced important insights into neural network internals, the field has yet to establish a standardized system to audit experiments.

By Michael Lan, Narmeen Fatimah Oozeer, Chaithanya Bandi, Philip Quirke, Austin Meek, Fazl Barez, Amirali Abdullah
arXiv AI
Aug 17

ASSERT: A Measurement Pipeline for GenAI Audits

arXiv:2608. 13840v1 Announce Type: cross Abstract: Audits of generative AI (GenAI) systems often summarize behavior as a reported rate: how often the audited system complies with policy.

By Riccardo Fogliato, Abhinav Palia, Xiawei Wang, Emily Sheng, Chad Atalla, Jean Garcia-Gathright, Nicholas Pangakis, Sharman Tan, Dan Vann, Hannah Washington, P. Alex Dow, Heba Elfardy, Hanna Wallach, Sandeep Atluri
arXiv AI
Sep 17

Who Audits Whom, on What Substrate, with What Evidence? An Independence-Graded Audit Protocol for Agentic AI

The paper proposes an independence‑graded audit protocol for agentic AI systems, arguing that independence should be evaluated along three orthogonal axes: principal independence, substrate independence, and evidence independence. It introduces a seven‑step protocol based on the beta‑factor model from reliability engineering, demonstrates its application through a structural detectability analysis and a Monte Carlo study, and maps the framework to relevant regulatory standards such as the EU AI Act, ISO/IEC 42006, and UK public‑sector guidance.

By Mohamed Chahine Ghanem