arXiv AI

CIRCLE: A Framework for Evaluating AI from a Real-World Lens

arXiv AI
Sep 1

Operationalising AI Regulatory Sandboxes: Activities, Requirements, and Technical Assessment under the EU AI Act

The paper "Operationalising AI Regulatory Sandboxes: Activities, Requirements, and Technical Assessment under the EU AI Act" outlines a detailed framework for implementing AI Regulatory Sandboxes (AIRS) under the EU AI Act. It maps the sandbox lifecycle into 29 activities, distinguishes between a Core AIRS and an Extended AIRS that includes an AI Technical Sandbox (AITS), and derives 15 infrastructural and governance requirements linked to these activities and provider obligations. The authors also introduce the Sandbox Configurator, an open‑source tool to instantiate AITS environments, aiming to provide structured workflows for regulators, robust evaluation methods for experts, and a transparent compliance pathway for AI providers.

By Alessio Buscemi, Thibault Simonetto, Daniele Pagani, German Castignani, Maxime Cordy, Jordi Cabot
arXiv AI
Sep 18

A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems

The paper introduces a unified evaluation framework for assessing the trustworthiness of large language models, agentic AI, and multimodal systems. It connects output-level, trajectory-level, and cross-modal assessments across eight dimensions—capability, robustness, safety, fairness, transparency, governance, oversight, and efficiency—while preserving system-specific metrics and providing uncertainty estimates. A meta-evaluation layer checks the validity, reliability, and reproducibility of the evaluation itself, and the framework aligns with governance standards and regulatory requirements.

By Shaina Raza, Ahmed Y. Radwan, Imran Liaquat, Kathryn Hume
arXiv AI
Sep 25

The Gold in Bias: Maturing the AI Design Process through Verification

The paper proposes rethinking bias in AI as a diagnostic tool rather than merely a flaw to be minimized. It introduces a multidimensional framework that examines bias across origin, lifecycle emergence, technical causes, and validation methods, covering 30 bias types, 16 verification methods, and 20 countermeasures for both traditional and generative AI. The authors present a hierarchical evidence framework distinguishing internal and external validity, and advocate for Ethics by Design principles to embed bias verification throughout the AI development lifecycle.

By Samira Maghool, Paolo Ceravolo
arXiv AI
4d ago

Rethinking Data Quality for AI-Driven Systems: Evidence from Practitioner Interviews

The study examines how practitioners in AI-driven systems define, assess, and manage data quality, revealing six key themes. It highlights shifts in traceability, the use of models as quality assessors, and the emergence of new data objects such as agent context and synthetic data. The research proposes a lifecycle assurance framework to provide evidence that data supports specific AI claims throughout model behavior, judgments, and agent actions.

By Hariharan Gopinath, Jan Bosch, Helena Holmstr\"om Olsson
arXiv AI
Jun 2

Comprehensive AI governance requires addressing non-model gains

arXiv:2606. 00047v1 Announce Type: cross Abstract: Frontier AI governance often centres on the model-level governance paradigm, which assumes that a model's capability profile is primarily a function of the compute and data used during training.

By Arthur Goemans, Dan Altman, Noemi Dreksler, Jonas Freund, Milan Gandhi, Zhengdong Wang, Sarah Cogan, Sebastien Krier, Demetra Brady, Lewis Ho, Allan Dafoe
arXiv AI
Aug 5

AI Sandbox: Technical Report

arXiv:2608. 02679v1 Announce Type: cross Abstract: Collaborative AI experimentation across industry and academia requires platforms that enable rapid prototyping while preserving controlled access, tenant separation, and transparent workflows.

By Muhammad Waseem, Md Aidul Islam, Md Nasir Uddin Shuvo, Md Mahade Hasan, Kai-Kristian Kemell, Jussi Rasku, Mika Saari, Vilma Saari, Roope Pajasmaa, Markku Oivo, Pekka Abrahamsson