The paper introduces a framework for auditing grounding claims in AI systems, evaluating how tokens like "cat" are linked to real-world concepts. It reports measurements, evidence, and verdicts based on explicit acceptance criteria, assessing accuracy, robustness, and composition while documenting how mechanisms were acquired and retained. A toy gridworld example shows an agent correctly interpreting individual symbols but failing a withheld combination, highlighting a deviation from the declared composition rule and demonstrating the audit’s ability to pinpoint such issues.
arXiv:2606. 03031v1 Announce Type: new Abstract: Structured financial audit verification is difficult for language-model agents because correctness depends on structured evidence rather than text alone.
arXiv:2609.39228v1 Announce Type: new Abstract: We present FYAN, a human--AI harness for document-level mathematical formalization. Rather than treating theorems in isolation, FYAN coordinates an end...
The paper introduces CORA (Counterfactual, Observable Redundancy Audit), a protocol for auditing website redundancy by measuring repetition load, normal-use tax, and failure-domain recovery reserve. Each audit run records screenshots, stable element identities, and task traces, while a versioned vision‑language model generates annotations that are validated and released only if they meet calibrated criteria. Experiments on a transparent mechanistic testbed show that CORA’s factorized representation separates reserve from normal-use tax and predicts perturbed success more accurately than scalar-load baselines, but it withholds automated scores when instruments fail to meet release requirements, indicating that CORA is an auditable candidate procedure for the studied benchmark rather than a universal standard.
arXiv:2608. 07688v1 Announce Type: new Abstract: IT audits require auditors to judge whether heterogeneous organizational evidence satisfies semantic security and compliance controls.
arXiv:2607. 08288v1 Announce Type: cross Abstract: In critical infrastructure, operational technology environments often cannot be actively scanned, and yet active system feedback is needed for risk assessment and compliance.
arXiv:2607. 08028v1 Announce Type: new Abstract: Enterprise large language model (LLM) applications often begin as prototypes whose behavior is carried by prompts and retrieval context.
arXiv:2608.23370v1 Announce Type: new Abstract: Large Language Models (LLMs) recognise patterns but do not natively track the path of exclusions that a coherent discourse demands. When an input rests...
CARGO is a framework for evaluating agentic AI systems in production that addresses the problem of reference-instance divergence (RID), where reference-based judges penalize correct answers that involve different entity identifiers. It treats retrieved references as procedural exemplars, grounds judgments in the live instance’s context, assigns a three-way status to claims, and gates evaluation by retrieval confidence. Using the CARGO-Bench diagnostic suite, CARGO eliminates false penalties and improves discrimination while revealing a limitation in detecting procedural corruptions.
LOGIC is a benchmark and evaluation framework that tests how language models can ground engineering requests in a deterministic inventory of candidate changes before propagating selected changes through an electrical traceability graph. The benchmark includes 168 scenarios—144 for selection and 24 for abstention—and evaluates three 7–8B models against intent‑agnostic, lexical, and structured‑evidence methods. Results show that structured evidence can achieve perfect candidate F1 on anchored cases, while large language models perform better on relational‑paraphrase cases; however, grounding accuracy drops as candidate inventories grow, and strict evidence gating reduces false positives but may also remove correct selections.
arXiv:2609.15319v1 Announce Type: cross Abstract: Frontier models score well on shallow document/chart reading tasks. In a controlled data-room audit, moving evidence into buried conditions reduced a...
arXiv:2608.22160v1 Announce Type: new Abstract: Physical automation is scaling toward fleets of embodied machines commanded by an AI brain. Early deployments already run factories and warehouses at p...
arXiv:2608. 16852v1 Announce Type: new Abstract: Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy.