← Back to all news
arXiv Computer Vision By Ge Kong, Yongtong Cao

From Subjective Judgments to Auditable Standards:Protocol-Guided AI Auditing of Website Redundancy

Read the original on arXiv Computer Vision →

The paper introduces CORA (Counterfactual, Observable Redundancy Audit), a protocol for auditing website redundancy by measuring repetition load, normal-use tax, and failure-domain recovery reserve. Each audit run records screenshots, stable element identities, and task traces, while a versioned vision‑language model generates annotations that are validated and released only if they meet calibrated criteria. Experiments on a transparent mechanistic testbed show that CORA’s factorized representation separates reserve from normal-use tax and predicts perturbed success more accurately than scalar-load baselines, but it withholds automated scores when instruments fail to meet release requirements, indicating that CORA is an auditable candidate procedure for the studied benchmark rather than a universal standard.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv AI
Aug 26

Confidently Wrong, Silently So: Auditing Undetectable Failures of a Deployed On-Device Language Model

The paper audits a developer‑accessible on‑device language model, revealing that it can confidently produce incorrect answers while refusing benign prompts, a phenomenon termed task‑asymmetric miscalibration. The model’s confident outputs are surface‑indistinguishable, with classifiers based on user‑visible features failing to separate correct from wrong responses. The authors propose a model‑agnostic audit protocol, a surface‑indistinguishability test, and a black‑box consistency wrapper that improves reliability without requiring model access.

By Shashwat Pandey, Satwik Pandey, Suresh Raghu
llmssafety
More like this →
arXiv Computer Vision
Oct 12

Auditing VLM Benchmarks from Existing Evaluation Runs

The paper introduces an audit layer for vision‑language model benchmarks that reuses per‑question artifacts from evaluation runs to assess the benchmark itself. Six detectors analyze saved outputs, and an additional text‑only rerun removes visual inputs to produce per‑question reports and aggregate statistics, including permutation‑null excess and interval estimates. Applying this method to the MMStar benchmark reveals that 20.2% of its questions show no accuracy gain from images, with a small but significant excess in mathematics and science categories, and identifies questions where all models chose the same non‑gold option.

By Denis Degtiarev, Anna Gromova, Sergey Katkov, Artem Mikhailin, Safina Mukhamedshina, Albert Nasybullin, Karina Shaikhutdinova, Artem Tuzov
llmsmultimodalbenchmarks
More like this →