arXiv AI

What Will Remain Human in Software Architecture? A Focus Group Report

The report examines how software architects view the growing use of AI development agents in their field. A focus group of 22 industry and academic participants discussed current practices, trust, validation, governance, and educational implications, concluding that decision‑making, accountability, and guardrail authoring remain human responsibilities. They introduced the concept of harness engineering—building systems that govern AI‑assisted creation—and identified criticality and cognitive debt as key factors for calibrating human oversight.

arXiv AI
6d ago

Key Institutional Considerations for Designing Effective Independent Verification Organizations

The paper discusses the growing role of Independent Verification Organizations (IVOs) in evaluating AI technologies, noting that while consensus exists on operating conditions, there is a lack of discourse on institutional and policy specifications. It argues for IVO designs that can assess AI models, systems, and applications across both frontier and enterprise contexts, extending beyond catastrophic risks to everyday risks. Drawing on institutional examples from finance, education, public health, and science, the authors outline key considerations for IVOs, including their target audiences, required access, interactions with governing structures, funding, and integration into the broader assurance ecosystem.

By Uma Kalkar, Rumman Chowdhury, Reva Schwartz, Gabriella Waters
arXiv AI
6d ago

Understanding-First AI Engineering: Purposes, Reference Architecture, and Architectural Patterns for Human Understanding

The paper proposes an "understanding-first AI engineering" approach that prioritises human understanding as a core design principle for AI systems. It categorises purposes of understanding into reflective agency, effective action, collective governance, and epistemic value, and presents a reference architecture linking production, verification, use, and learning through shared records. Five architectural patterns—parallel learning, explanatory models, goal exploration, audience‑specific views, and distributed expertise—are suggested to guide design choices, with scenario analyses illustrating their application and limitations.

By Liming Zhu, Qinghua Lu, Ming Ding
arXiv AI
Sep 1

Operationalising AI Regulatory Sandboxes: Activities, Requirements, and Technical Assessment under the EU AI Act

The paper "Operationalising AI Regulatory Sandboxes: Activities, Requirements, and Technical Assessment under the EU AI Act" outlines a detailed framework for implementing AI Regulatory Sandboxes (AIRS) under the EU AI Act. It maps the sandbox lifecycle into 29 activities, distinguishes between a Core AIRS and an Extended AIRS that includes an AI Technical Sandbox (AITS), and derives 15 infrastructural and governance requirements linked to these activities and provider obligations. The authors also introduce the Sandbox Configurator, an open‑source tool to instantiate AITS environments, aiming to provide structured workflows for regulators, robust evaluation methods for experts, and a transparent compliance pathway for AI providers.

By Alessio Buscemi, Thibault Simonetto, Daniele Pagani, German Castignani, Maxime Cordy, Jordi Cabot
arXiv AI
Aug 26

AI Agents Push Humans Out of the Loop

AI agents are increasingly autonomous, posing significant risks that current designs hinder effective human oversight. The paper argues that oversight is degraded by both design choices and the cognitive decline of users who rely heavily on automation. It calls for prioritizing human cognitive needs in AI agent development, proposing design affordances and protocols to maintain critical judgment and counter skill atrophy.

By Margaret Mitchell, Avijit Ghosh, Samir Passi
arXiv Machine Learning
Sep 11

From Protocols to Evidence: Bounded Claims for AI in Service of the Common Good

The paper argues that AI should be evaluated not only by principles but by concrete protocols that translate commitments into roles, requirements, records, oversight, and assessment. It introduces a rupture test linking institutional baselines to system evaluation, and distinguishes evidence‑bounded deployment from measurement‑bounded governance. The authors propose the RISE AI architecture to make bounded, evidence‑based claims about Responsibility, Inclusivity, Safety, and Empowerment, emphasizing the need for engineering, institutional repair, and ongoing moral judgment.

By Nitesh V. Chawla, Paulo Benanti