AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,884 stories · RSS feed

arXiv Statistics ML
Aug 25

A Commutator Framework for Selective Spectral Alignment in Deep Neural Networks

The paper introduces a finite‑width geometric framework that explains how learned feature geometries are organized, transported, and selectively aligned in deep neural networks. It quantifies incompatibilities among weight‑generated covariance, gates, and backward sensitivities using three families of commutators, and provides exact layerwise identities that decompose these commutators into sources such as downstream transport, adjacent‑layer imbalance, and nonlinear gate‑covariance interactions. The study demonstrates that spectral alignment is a layer‑ and scale‑dependent compatibility phenomenon governed by transport, interaction, cancellation, and possible damping, rather than a universal consequence of training.

By Kaj Nystr\"om
arXiv AI
Aug 25

A Survey on Human-AI Collaboration with Large Foundation Models

The paper surveys how Large Foundation Models (LFMs) can be integrated into Human‑AI Collaboration (HAI) to enhance problem‑solving and decision‑making. It outlines four key areas—human‑guided model development, collaborative design principles, ethical and governance frameworks, and high‑stakes applications—while emphasizing that effective HAI systems arise from careful, human‑centered design rather than merely stronger models. The survey also identifies open challenges related to safety, fairness, and control, aiming to guide future research toward reliable, trustworthy, and beneficial LFM‑based partnerships.

By Vanshika Vats, Marzia Binta Nizam, Minghao Liu, Ziyuan Wang, Richard Ho, Mohnish Sai Prasad, Vincent Titterton, Sai Venkat Malreddy, Riya Aggarwal, Yanwen Xu, Lei Ding, Jay Mehta, Nathan Grinnell, Li Liu, Sijia Zhong, Devanathan Nallur Gandamani, Xinyi Tang, Rohan Ghosalkar, Celeste Shen, Rachel Shen, Nafisa Hussain, Kesav Ravichandran, James Davis
arXiv AI
Aug 25

Robust Multi-Agent Reinforcement Learning for Small UAS Separation Assurance under GPS Degradation and Spoofing

The paper presents a robust multi‑agent reinforcement learning framework for small unmanned aircraft systems (sUAS) to maintain separation assurance when GPS data is degraded or spoofed. By modeling state observation corruption as a zero‑sum game, the authors derive a closed‑form adversarial perturbation that eliminates iterative inner optimization and can be evaluated in linear time. Integrating this perturbation into a policy‑gradient MARL algorithm yields a counter‑policy that achieves near‑zero collision rates in high‑density simulations even with up to 35% observation corruption, outperforming non‑adversarial baselines.

By Alex Zongo, Filippos Fotiadis, Ufuk Topcu, Peng Wei
arXiv AI
Aug 25

Hyperbolic Hierarchical Clustering for Visual Representation Learning

The paper introduces ClusterMixer, a token mixer based on hierarchical clustering in hyperbolic space, designed to be transparent and interpretable. It forms the core of a new vision backbone called HCFormer, which incorporates multiple clustering strategies to maintain strong performance. Experiments show HCFormer surpasses existing backbones on tasks such as image classification, object detection, instance segmentation, and semantic segmentation.

By Jianan Wei, Guikun Chen, Zhiyuan Weng, Chunchao Guo, Yujia Wang, Wenguan Wang
arXiv AI
Aug 25

Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms

The paper introduces BanglaSafe, a benchmark of 879 Bengali prompts that covers 17 culturally grounded harm categories and five prompting conditions. Evaluation of 18 frontier LLMs shows that 53.6% of responses are unsafe or partially unsafe, with 14.7% containing strictly harmful content. The study finds that the writing style within Bengali has a stronger impact on safety than the language switch itself, and that current safety classifiers struggle to reliably evaluate Bengali content.

By Naymul Islam, Nusrat Jahan Lia, Shubhashis Roy Dipta, Sabik Bin Sultan, Abdullah Khan Zehady
arXiv AI
Aug 25

A New Type of Adversarial Examples

The paper introduces a new class of adversarial examples that are markedly different from original inputs yet produce the same model output. It presents algorithms such as NI-FGSM, NI-FGM, and their momentum variants (NMI-FGSM, NMI-FGM) to generate these examples. The authors demonstrate that these adversarial examples are not confined to the vicinity of training data but are spread throughout the sample space.

By Xingyang Nie, Caoliang Zhang, Su Pan, Biao Wang, Huilin Ge, Tao Fang
arXiv AI
Aug 25

ONOTE: Hypergraph-Grounded Omnimodal Reasoning for Computational Music Science

ONOTE is a unified framework that treats music as a scientifically structured domain of measurable cross-representation correspondences, focusing on omnimodal notation processing centered on sheet music. It introduces a test-only benchmark drawing from diverse musical sources—including staff, Jianpu, and tablature—across varied genres, instruments, and structural conditions, with aligned multimodal derivatives. The framework supports four complementary tasks—score understanding, notation conversion, audio transcription, and symbolic generation—while constructing a provenance-bearing proposition hypergraph from external music-theory materials for evidence retrieval and deterministic validity checks.

By Menghe Ma, Siqing Wei, Yuecheng Xing, Ziyue Zhu, Zhenghong Lin, Yaheng Wang, Fanhong Meng, Peijun Han, Luu Anh Tuan, Haoran Luo
arXiv AI
Aug 25

SDoH-Aware Narrative Anchoring Bias in Medical LLMs for Trustworthy Clinical Decision Support

The paper investigates how medical large language models (LLMs) may exhibit narrative anchoring bias when presented with the same clinical case in different patient voices. Using the NarrativeShield SDoH MedQA dataset, the authors evaluate three Qwen2.5 instruction‑tuned LLMs (1.5B, 3B, 7B) on 300 clinical cases, reporting metrics such as persona‑level accuracy, counterfactual consistency, correct consistency, and narrative sensitivity error. The 7B model achieves the highest accuracy (56.33 %) and correct consistency (40.33 %), yet narrative sensitivity errors remain substantial (31.67 %).

By Ahnaf Atef Choudhury, Ramkrishna Saha
arXiv Computation and Language
Aug 25

DIAG: Diagnostic Iterative Alignment and Generation for Data-Efficient Mathematical Preference Distillation

DIAG is a Diagnostic Iterative Alignment and Generation framework designed to improve data efficiency in aligning large language models for mathematical reasoning. It adaptively reshapes the practice distribution by first diagnosing valid preference-pair yield to calibrate exploration and exploitation, then generating targeted practice from the model’s failure traces. The approach is theoretically framed as a teacher‑mediated approximation to KL‑regularized reweighting, and experiments show that DIAG increases preference-pair yield and reasoning performance under the same training budget.

By Guhan Chen, Songtao Tian, Bohan Li, Hejin Wang, YeXin Xie, Zixiong Yu
arXiv Computer Vision
Aug 25

Reduce the Artifacts Bias for More Generalizable AI-Generated Image Detection

The paper proposes Artifact-Complementary Expert Fusion (ACEF), a two‑stage framework that enhances AI‑generated image detection by combining two types of reconstruction artifacts—VAE/DDIM and SRGAN—into aligned synthetic negatives. ACEF first builds artifact‑specific experts using LoRA adaptation on a frozen backbone, then fuses their multi‑layer evidence with Layer‑wise Artifact‑Complementary Fusion (LACF) to mitigate conflicts between artifact manifolds. Experiments on 13 benchmarks show that this approach improves generalizability over existing state‑of‑the‑art methods.

By Yiheng Li, Yang Yang, Wenhao Wang, Zichang Tan, Zecheng Lin, Li Gao, Zhen Lei
arXiv Computation and Language
Aug 25

The Communication Map of a Transformer

The paper introduces the Communication Map, a method that charts every potential communication channel in a transformer model using only its weights. It generalizes previous coupling metrics into a single coefficient covering all 18 connection classes, revealing that 70‑89% of head pairs are non‑randomly oriented and identifying strong or avoiding couplings. The authors demonstrate the map’s utility by recovering known induction circuits and uncovering a two‑dimensional stream subspace whose removal eliminates induction capabilities across several models.

By Richard Zhe Wang
arXiv AI
Aug 25

TEE-X: TEE-aware Acceleration Framework for Large Vision Models at the Edge

TEE-X is a TEE‑aware acceleration framework designed to run large vision models, such as Vision Transformers, entirely within Trusted Execution Environments. It introduces a sensitivity‑aware modularization technique and vectorization to overcome memory constraints and latency challenges on edge devices. The framework is validated on OP‑TEE for Arm TrustZone and optimized for the NVIDIA Jetson AGX Xavier, achieving GPU‑level inference latency with minimal accuracy‑latency trade‑offs.

By Kurt M Wilson, Mohaiminul Al Nahian, Abeer Matar A. Almalky, Sadat Shahriyar, Souvik Kundu, Zhishan Guo, Abdullah Al Arafat, Adnan Siraj Rakin