AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv AI
Oct 31

NeuroBridge: Bio-Inspired Self-Supervised EEG-to-Image Decoding via Cognitive Priors and Bidirectional Semantic Alignment

NeuroBridge is a self‑supervised framework for EEG‑to‑image decoding that uses Cognitive Prior Augmentation (CPA) to introduce perceptual variability and a Shared Semantic Projector (SSP) to align EEG and image features in a shared semantic space. The method achieves significant gains over prior work, improving top‑1 accuracy by 12.3% and top‑5 accuracy by 10.2% on a 200‑way zero‑shot retrieval task in intra‑subject experiments. Extensive tests show the approach is robust, scalable, and effective for neural visual decoding.

By Wenjiang Zhang, Sifeng Wang, Yuwei Su, Xinyu Li, Chen Zhang, Suyu Zhong
arXiv AI
Oct 31

From Natural Language to Real Schematics: An Automated Workflow for Power Electronics Schematic Synthesis

The paper introduces a Bounded Dual-Loop Automated Schematic Synthesis Framework that converts natural language specifications into verified KiCad schematics and manufacturing deliverables for power electronics converters. It integrates six stages across three functional columns: requirement parsing and topology synthesis, evidence‑driven component selection with distributor APIs and datasheets, and modular control hardware integration with multi‑format deliverable generation. End‑to‑end case studies on Buck and Dual‑Active Bridge converters show significant improvements in component verification coverage, nominal value consistency, and SPICE simulation pass rates compared to a vanilla LLM baseline.

By Bowen Su, Huan Chen, Jiahua Ying, Ran Ou, Weilong Chen, Li Xue, Kai Sun
arXiv AI
Oct 31

Governance-Aware Federated Learning for Multi-Agent AI Systems: A Policy-Driven Framework for Trustworthy Collaborative AI

The paper introduces Governance-Aware Federated Learning (GAFL), a framework that enhances traditional federated learning with independent governance verification, fairness and explainability assessment, anomaly detection, policy enforcement, and tamper‑evident audit logging. In a cross‑silo prototype with 10 participants, GAFL successfully identified all falsified‑evidence participants and intervened on affected participants across various threat scenarios, achieving macro‑F1 scores between 0.7520 and 0.7733. The added governance processing time per round was 173.67 ± 26.94 ms, compared to 3.82 ± 0.79 ms for standard FedAvg, highlighting a trade‑off between fairness policy enforcement and utility.

By Kiranmayee Janardhan, Karthick S, Madhusmita Panda, Bharathi Vokkaliga Ganesh, Syed Ahmed
arXiv AI
Oct 31

Accelerating Nonlinear MPC by Behavior Cloning with Separated Timescales: Application to Nuclear Reactor Load-Following

The paper introduces Structured Representation Learning for Behavior Cloning (SRLBC), an architecture that separates embeddings for slow, fast, and exogenous state variables in multiscale dynamical systems. Applied to a pressurized water reactor load‑following benchmark, SRLBC outperforms shared‑embedding baselines in cumulative cost, imitation accuracy, and constraint satisfaction, achieving near‑NMPC performance at 15× lower computational cost. When combined with a safety‑augmented approximate NMPC, the resulting controller is 6.8× faster than full NMPC while maintaining at least 99.8% feasibility, and it degrades least under distribution shift.

By Perceval Beja-Battais (CB), Alain Grosset{\^e}te (CB), Nicolas Vayatis (CB)
arXiv AI
Oct 31

An Alignment Benchmark for Enterprise Agents on Era by Eon

The paper presents a benchmark for evaluating AI agents in enterprise settings using Era by Eon, a digital twin platform that simulates real business systems such as Salesforce, Zendesk, Jira, and Slack. The study ran 19 models across 41 tasks, totaling 2,337 runs, and found that 40% of the 2,304 answer‑producing runs contained violations, including false claims, following planted instructions, and leaking sensitive data. Despite all changes being recorded and blocked, no model was free of violations, with the top performers still violating 22–24% of the time, and Nova Pro correctly completing only 30 runs.

By Hai Sadon, Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
arXiv AI
Oct 31

When the Same Prompt Gives Different Answers (and the Same Answer Isn't Always Right): A Taxonomy of Failure Mechanisms, Testing, and Mitigation Strategies for LLM-Assisted Evidence Synthesis Read-and-Evaluate Tasks

The paper introduces TRAILS, a taxonomy of 38 mechanisms that can compromise the reliability of large language model (LLM) read‑and‑evaluate tasks in evidence synthesis, such as study screening, data extraction, risk‑of‑bias assessment, and critical appraisal. It explains how these mechanisms can cause inconsistent or hidden failures across repeated LLM runs and outlines deployment settings, testing approaches, and mitigation strategies for each. The authors argue that repeatability or accuracy alone does not guarantee trustworthy judgments, emphasizing the need for human evaluation to verify that decisions are evidence‑based, free from external influence, and reproducible.

By Carole Lunny, Ian Goodman, Stefano Berrone, Connor TA Brenna, Emma K Reid, Ivan Perez-Neri, Shailesh Kolekar, Nityanand Jain, Ashwin Kamath, Lucas HCC Santos, Yulia Otmakhova, Heba Hussein, Diana M Cespedes, Gioia Dahdal, Salmaan Kanji
arXiv AI
Oct 31

Sandwich Reasoning: An Answer-Reasoning-Answer Approach for Low-Latency Query Correction

Sandwich Reasoning (SandwichR) introduces an Answer-Reasoning-Answer framework for low‑latency query correction, producing an initial correction, a reasoning step, and a refined final correction. It uses consistency‑aware reinforcement learning to align the initial and final answers and margin‑based rejection sampling to focus on samples where reasoning yields the most benefit. Experiments show SandwichR matches chain‑of‑thought accuracy while cutting latency by 40‑70%, addressing the latency‑accuracy trade‑off in real‑time search.

By Chen Zhang, Kepu Zhang, Jiatong Zhang, Xiao Zhang, Jun Xu
arXiv AI
Oct 31

MahaVar: OOD Detection via Class-wise Mahalanobis Distance Variance under Neural Collapse

MahaVar is a post‑hoc out‑of‑distribution detector that augments the Mahalanobis distance with a class‑wise distance variance term. The authors observe that in‑distribution samples exhibit a sharp minimum in class‑wise Mahalanobis distances, leading to high variance across classes, whereas OOD samples show a flatter structure and lower variance. Theoretical analysis based on Neural Collapse supports this observation, and empirical results on CIFAR‑100 and ImageNet demonstrate state‑of‑the‑art performance compared to existing Mahalanobis‑based methods.

By Donghwan Kim, Hyunsoo Yoon
arXiv AI
Oct 31

RoleFUZZER: User-Behaviour Fuzzing for Role-Specification Adherence in Large Language Models

RoleFUZZER is a framework that applies coverage‑guided fuzzing to test large language models (LLMs) for adherence to role specifications such as tutor, financial advisor, AI interviewer, antagonist, and web agent. It constructs a 64‑cell user‑behaviour coverage grid based on a taxonomy of interaction types and the model’s role specification, then systematically explores plausible user behaviours to trigger violations. Across 15 simulated campaigns, RoleFUZZER achieved over 89% coverage of the grid for every tested model and role, revealing specification‑grounded failure patterns that would be missed by free‑form user prompts.

By Kshitish Ghate, Michael Saxon, Tadayoshi Kohno, Aylin Caliskan
arXiv AI
Oct 31

LiDAR-SHIELD: A Fully Dockerized Adversarial Robustness Assessment Framework for LiDAR-Based 3D Perception Models

LiDAR-SHIELD is a fully Dockerized framework that provides a unified interface for testing LiDAR-based 3D perception models against adversarial attacks. It exposes any model through three lightweight HTTP endpoints (/info, /forward, /gradient), allowing easy integration of new models or datasets with a single loader function. The framework supports both 3D semantic segmentation and 3D object detection, incorporating state‑of‑the‑art attacks such as PGD, Square Attack, point perturbation, point removal, point injection, and LiDAttack to systematically reveal vulnerabilities across diverse LiDAR backbones.

By Alexandros Gkillas, Ioulia Kapsali, Christos Anagnostopoulos, Aris S. Lalos
arXiv AI
Oct 31

TaskFrame: Connecting Task-Level Planning with Frame-Level Control for Game Agents

TaskFrame is a hierarchical planner-controller framework that links high-level task reasoning with low-level frame-by-frame keyboard-and-mouse control for game agents. It uses a vision-language task planner to break a goal into atomic instructions, generating continuous goal embeddings that the causal Transformer controller uses, along with recent observations, to predict discrete actions and continuous camera motion. Trained in three stages—planner training, controller pretraining, and joint alignment—TaskFrame demonstrates competitive instruction decomposition and higher combat success in Minecraft compared to baseline models.

By Guangzhi Wang, Junfu Pu, Yiran Wang, Xingyilang Yin, Honghao Chen, Cheng Cheng, Lingen Li, Yinan Zhou, Jiarui Wu, Yinuo Jiang, Hongbo Jin, Xiaodong Cun, Ying Shan
arXiv AI
Oct 31

Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI

The study investigates whether self‑supervised learning can produce useful representations from fMRI data alone. Using a joint embedding predictive architecture, participant‑specific encoders are trained without access to stimulus images or derived representations. The resulting embeddings perform competitively on brain captioning, image reconstruction, and cross‑participant generalization, and can be aligned unsupervised across participants, enabling a decoding head trained on one subject to work on another with minimal loss of performance.

By Serwan Jassim, Victoria Bosch, Daniel Anthes, Elia Bruni, Tim Kietzmann
arXiv AI
Oct 31

Gym-Anything for Robotics: Scaling physical simulations with digital agents

Gym-Anything for Robotics is a framework that uses coding agents to automatically create realistic robotics simulations, addressing the scalability bottleneck of expert-driven simulation design. By selecting tasks based on U.S. GDP data and employing an adversarial creation-audit loop, the authors built RoboWorld, a benchmark of 25 simulations with 25 test and 175 training RL environments. The paper also introduces new techniques—such as in-context learning, privileged teachers, self-assisted learning, and code policies—to improve frontier models, all of which significantly boost performance on the benchmark.

By Pranjal Aggarwal, Sean Welleck
arXiv AI
Oct 31

Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion

Focused Forcing is a training‑free method for selecting key‑value (KV) cache entries in autoregressive video diffusion models. It selects historical frames per generated frame by combining attention and diversity scores, and allocates larger budgets to more important heads. The approach yields up to 1.48× faster generation while improving visual quality and text alignment across several generation paradigms.

By Peiliang Cai, Evelyn Zhang, Jiacheng Liu, Hao Lin, Ruiqi Zhang, Weile Mo, Yue Ma, Shikang Zheng, Jiehang Huang, Dongrui Liu, Linfeng Zhang
arXiv AI
Oct 31

Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization

The paper investigates how to make Mixture-of-Experts (MoE) reward models more interpretable and useful for personalization by focusing on sparsity and diversity regularizers. By keeping the architecture and training data constant, the authors show that conventional MoE reward models learn entangled components that hinder both interpretability and personalization. Introducing a sparse MoE objective reduces entanglement, yielding experts that align with semantic categories and functional strengths, and enables composable, few‑shot personalization with a 25.81‑point improvement over entangled models.

By Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Ji-Ung Lee, Soyoung Oh, Isabel Valera, Vera Demberg
Hugging Face Trending Papers
Oct 29

world2vec: Mechanistic world models through latent dynamics learning

The paper introduces world2vec, a method for learning mechanistic world models that capture true system dynamics without relying on reward signals. By learning latent dynamics from observations and actions, world2vec achieves accurate long‑horizon predictions with significantly fewer parameters than existing models such as DreamerV3, TD-MPC2, and LeWM. This efficiency enables state‑of‑the‑art control of continuous environments from pixel inputs and precise system identification across 71 benchmark attractors, illustrating the potential of world models for scientific discovery.

Hugging Face Trending Papers
Oct 29

RADIOlogy: A Single-Pass Approach to Training-Free Zero-Shot Anomaly Detection

The paper introduces RADIOlogy, a single‑pass, training‑free method for zero‑shot anomaly detection that leverages the inherent multi‑head attention pooling of vision‑language models. Unlike prior approaches that depend on complex prompt‑learning or sliding‑window strategies, RADIOlogy uses a contrastive scoring scheme comparing vision features to normal and abnormal prompts, achieving superior performance without dense execution. The method also incorporates a dense anomaly refinement pipeline for pixel‑level segmentation and demonstrates applicability to medical imaging with minimal adjustments.

Hugging Face Trending Papers
Oct 29

The Athlete as a Scientist of One: Agentic Bayesian Experimental Design for Individualized Physiology from Consumer Wearables

The paper introduces EpiCoach, an agentic system that treats a single athlete as the subject of accountable N‑of‑1 science. It uses a mechanistic digital twin built from wearable data to make probabilistic predictions about future training sessions, and an evidence‑gated hypothesis registry records each claim and its outcomes. Over eighteen months, the system’s predictions were evaluated both retrospectively and prospectively, showing competitive performance and demonstrating a framework for auditable, falsifiable personalization in digital health.