arXiv AI
Jun 30
Defeat Devices in AI Systems
arXiv:2606. 28863v1 Announce Type: cross Abstract: AI systems increasingly exhibit behavior that differs systematically between evaluation and deployment contexts.
When Compliance Data Masquerades as Evaluation: Measurement Validity for Deployed AI Systems
Related stories
arXiv AI
4d ago
AI Deployment Accountability Engineering: A Vision for Accountable AI in Safety-Critical Socio-Technical Systems
The paper proposes AI Deployment Accountability Engineering (ADAE), a new subdiscipline focused on establishing measurable, continuous, and actionable accountability for AI systems once they are deployed. ADAE treats accountability as a deployment-layer property, aiming to ensure systems remain within acceptable risk limits, identify failure contexts, attribute failures across technical and human components, and translate technical failures into downstream consequences. The authors outline a research agenda built around four pillars—structured discovery of context-dependent failure modes, privacy-preserving accountability measurement, system-level risk analysis for agentic AI, and translation of technical failures into operational and institutional risks—to support timely intervention in safety-critical socio-technical environments.
arXiv AI
Aug 6
A Chain Is Only as Strong as Its Weakest Link: A Scoping Review of System Integration Audits in AI
arXiv:2608. 04921v1 Announce Type: cross Abstract: As AI systems become increasingly integrated into diverse interfaces and applications, model-centric audits are insufficient to address risks arising from interactions among system components and deployment environments.
arXiv AI
Aug 5
Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation
arXiv:2608. 00794v2 Announce Type: replace Abstract: Agentic AI evaluation pipelines produce benchmark scores that justify deployment decisions, safety certifications, and regulatory compliance claims.
arXiv AI
Jul 16
Post-Deployment Accountability in AI Governance: A Cross-Regulatory Empirical Analysis of AI Incidents
arXiv:2605. 16281v2 Announce Type: replace-cross Abstract: Post-deployment accountability has become central to AI governance, yet little empirical evidence shows whether monitoring, incident reporting, and impact assessment obligations are visible when AI systems fail.
arXiv AI
Aug 3
Beyond Component Testing: Validating Agentic AI Systems
arXiv:2607. 29405v1 Announce Type: new Abstract: Agentic AI systems act through multi-step trajectories that combine planning, tool use, memory, interaction, and adaptation.
arXiv AI
Aug 11
UGAF-ITS: A Standards Harmonization Framework and Validation Tool for Multi-Framework AI Governance in Distributed Intelligent Transportation Systems
arXiv:2604. 22789v2 Announce Type: replace-cross Abstract: Organizations deploying AI-enabled Intelligent Transportation Systems face fragmented governance: ISO/IEC~42001 demands a certifiable management system, the EU AI Act imposes binding high-risk obligations from August~2026, and the NIST AI Risk Management Framework structures voluntary practice.
arXiv AI
1d ago
Governance-as-Code: Translating EU AI Act Technical Requirements into Executable Compliance Pipelines for Generative AI Systems
The paper introduces Governance-as-Code (GaC), a framework that translates the EU AI Act’s technical requirements into 43 machine‑checkable acceptance criteria across six compliance modules. GaC runs within a CI/CD pipeline, producing Article‑indexed audit evidence and providing actual Rego policy code. The authors validate GaC on two enterprise deployments, showing it reproduces manual audit findings—including three penalty‑triggering violations—while reducing audit labor by about 75%.
arXiv AI
Jul 7
Open Problems in AI Incident Governance
arXiv:2607. 05163v1 Announce Type: cross Abstract: AI systems may produce failures after deployment that pre-deployment safety assessments do not anticipate.
arXiv AI
Sep 3
READY or Not: Reliable Enterprise Agent Deployment
READY or Not: Reliable Enterprise Agent Deployment introduces a framework for qualifying AI agents for enterprise workflows. It measures reliability and operating cost under various oversight policies, selects the minimum‑cost policy that meets a specified reliability target, and statistically qualifies it on held‑out cases. In a clinical audit study, READY revealed that two agents with nearly identical autonomous accuracy required markedly different levels of human review to achieve the same reliability target.
arXiv AI
Jun 30
EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures
arXiv:2606. 30219v1 Announce Type: new Abstract: LLM evaluation and AI safety face a shared measurement problem: benchmark scores, reward-model signals, and reported safety metrics can improve while the latent properties they are meant to represent remain difficult to verify.
arXiv Machine Learning
Aug 5
Evaluation Blindness: How Silent Measurement Failures Corrupt AI Systems from Training to Deployment
arXiv:2608. 02786v1 Announce Type: new Abstract: AI systems can fail silently.