The paper investigates how procedural traces—detailed step-by-step accounts of a language model’s reasoning—affect the decision-making of LLM overseers tasked with auditing another model’s outputs. Using signal detection theory, the authors evaluated five overseers on 19 compliance tasks, finding that while error detection remains high when disconfirming evidence is always visible, more elaborate traces shift the decision criterion toward rejection, leading to increased false alarms. The study also shows that providing option labels reduces the stated inability to link evidence to options, yet some overseers still exhibit residual rejection of correct work that grows with trace detail.
arXiv:2608. 16003v1 Announce Type: new Abstract: Automated checking pipelines increasingly place one language model as the checker and another (or the same one) as the fixer.
arXiv:2608.22160v1 Announce Type: new Abstract: Physical automation is scaling toward fleets of embodied machines commanded by an AI brain. Early deployments already run factories and warehouses at p...
The paper introduces the twin‑prefix framework to evaluate how the size of the verification unit—i.e., how many actions a pre‑execution LLM monitor reviews in one call—affects its performance. By pairing each gold plan with a twin that differs by a single write and injecting a controlled error, the authors isolate the impact of review length on catch rates and false rejections. Their findings show that longer review windows increase rejection rates but do not improve discrimination, with the highest informedness occurring at one or two actions across all judges and domains.
arXiv:2607. 02586v1 Announce Type: new Abstract: Governance frameworks ask AI providers and auditors for documented evaluation evidence, and perturbation-based construct-validity audits are a common form of that evidence.
arXiv:2608. 07688v1 Announce Type: new Abstract: IT audits require auditors to judge whether heterogeneous organizational evidence satisfies semantic security and compliance controls.
arXiv:2609.15319v1 Announce Type: cross Abstract: Frontier models score well on shallow document/chart reading tasks. In a controlled data-room audit, moving evidence into buried conditions reduced a...
LEDGER is a tracing and review system for large language model agents that constructs layered trace graphs from observed sessions. It groups raw trace records into Evidence Nodes and Workflow Nodes, anchors artifacts as evidence, and adds typed semantic edges linking claims to supporting actions, artifacts, and checks. The resulting traces reveal workflow decisions, artifact lineage, repair steps, validation coverage, and claim‑support paths for evidence‑centered audit.
arXiv:2606. 15127v1 Announce Type: new Abstract: Reasoning models are increasingly used in settings where the final answer is not the only object of review: educational tools may show students intermediate steps, decision-support systems may require human oversight, and audit workflows may inspect traces for misleading or biased input.
arXiv:2608. 13840v1 Announce Type: cross Abstract: Audits of generative AI (GenAI) systems often summarize behavior as a reported rate: how often the audited system complies with policy.
The paper introduces Governance-as-Code (GaC), a framework that translates the EU AI Act’s technical requirements into 43 machine‑checkable acceptance criteria across six compliance modules. GaC runs within a CI/CD pipeline, producing Article‑indexed audit evidence and providing actual Rego policy code. The authors validate GaC on two enterprise deployments, showing it reproduces manual audit findings—including three penalty‑triggering violations—while reducing audit labor by about 75%.
The paper introduces Policy-as-Skill (PaS), a modular runtime that packages evidence validation, review routing, version control, and auditability as executable, versioned policy capabilities for large language models. PaS was evaluated on 600 development tasks using a Gemma4 backend, achieving 53.8% exact accuracy and high audit completeness, and deterministic control further improves aggregate accuracy to 61.2% in a task‑dependent manner.
The paper "When Agents Act Unwatched: The Reduced‑Supervision Paradox in Agentic AI" discusses how the promise that AI systems will continue acting after users stop watching creates an accountability inversion. It argues that as stepwise supervision recedes, verification shifts into the runtime infrastructure—authority, records, interrupts, outcome checks, and repair—forming what the authors call the reduced‑supervision paradox. A 63‑artifact audit across research papers and engineering sources shows that agents’ action surfaces are more visible than the mechanisms needed to hold them accountable, with tool mediation and monitoring traces appearing in 40 and 37 artifacts, while checkpoint placement, validator independence, recovery, and contestability are rarely visible. "whyItMatters":"The study highlights that observable action paths can replace accountability when verification is moved onto users after meaningful intervention is no longer possible."