AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,638 stories · RSS feed

arXiv AI
1d ago

Multi-Resident Data Association in Smart Homes via MDP Modeling and Behavior Cloning

The paper introduces a new framework for multi-resident data association in smart homes, framing the problem as a Markov Decision Process and solving it with Behavior Cloning. Two policy architectures are proposed: a lightweight Transformer treating association as multi-label classification over event histories, and a fine-tuned Large Language Model that generates conditional text to leverage semantic knowledge. Experiments on three datasets show the approach outperforms existing baselines, especially for concurrent multi-resident events, and the LLM policy hints at cross-environment transfer potential.

By Xi Chen (M-PSI), Julien Cumin (LIG, M-PSI), Fano Ramparany (LIG, M-PSI), Dominique Vaufreydaz (LIG, M-PSI)
arXiv AI
1d ago

Using Experience Twice: Dynamic Process Rubrics for Privileged Token-Level Credit Assignment

The paper introduces Online Rubric Self-Distillation (ORSD), a method that extracts natural‑language process rubrics from agent trajectories and uses them as dense token‑level supervision signals. ORSD generates multiple rubric bundles per step, verifies them against trajectories, and selects the bundle that best aligns rubric satisfaction with task success. The selected rubric is then used to compute token‑specific process credit, which is added to the trajectory‑level advantage to enable fine‑grained credit assignment. Experiments on ALFWorld and WebShop demonstrate that ORSD speeds up policy learning and improves final performance for Qwen3 models within the same interaction budget.

By Shumao Wu, Kongcheng Zhang, Jianqiang Huang, Baisheng Lai
arXiv AI
1d ago

Addition by Subtraction: Toward Mitigating Fragile Safety Alignment in Large Language Models

The paper introduces a method called Addition by Subtraction to strengthen safety alignment in large language models. By gradually removing identified safety‑critical neurons during finetuning and re‑optimizing preferences, the authors reduce the model’s reliance on sparse components that can be targeted by pruning or ablation attacks. Experiments on four models from three vendors show that this approach improves robustness to both white‑box and black‑box jailbreaks, lowers attack success rates by up to 90%, and enhances truthfulness while preserving general capabilities.

By Hamid Dashtbani, Alexandra Dmitrienko
arXiv AI
1d ago

BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories

BEAKER is a new benchmark for evaluating embodied actionability, knowledge, experimentation, and reasoning in self‑driving chemical laboratories. It contains 1,000 expert‑curated question‑answer pairs across 11 image and video task types, covering affordance grounding, trajectory planning, equipment and state recognition, safety, apparatus setup, operation, step reasoning, and temporal procedure reasoning. The benchmark is manually created and reviewed by chemistry and embodied AI experts, and it tests 28 different multimodal large language models in an offline, single‑turn setting.

By Fei Lin, Tengchao Zhang, Ziyang Gong, Xiaotong Yu, Bohan Zhang, Yifan Zhou, Qihao Yang, Ji Dai, Dong Li, Yining Jiang, Qinghua Ni, Jun Huang, Qiang Zhang, Yue Zhou, Zhihang Zhong, Yonglin Tian, Fei-Yue Wang, Xue Yang
arXiv AI
1d ago

Behavior Graphs: Unsupervised Graph Structures for Interpreting Robot Deployment Data

The paper introduces behavior graphs, an unsupervised method that compresses high‑dimensional robot policy rollouts into a single directed graph. By clustering similar behavior modes, linking them temporally, and associating them with outcomes, the graphs provide an interpretable representation of deployment data. In a user study, participants preferred behavior graphs over raw videos, achieving comparable conclusions with less perceived effort, and across many datasets the graphs improved progress prediction accuracy.

By Erik Bauer, Jakob Thumm, Christopher Agia, Rohan Sinha, Chen Xu, Haruki Nishimura, Jeannette Bohg, Marco Pavone, Masha Itkina
arXiv AI
1d ago

When Does Sharing Work? Scaling a Shared SigLIP2 Across Image, Video, Audio, and Audio-Visual Representations

The paper introduces Omni-Encoder, a single SigLIP-2 Vision Transformer that jointly processes image, video, audio, and audio‑visual data using fully shared parameters. It replaces separate modality encoders and anchor‑based alignment with a unified architecture that tokenizes audio, applies multimodal rotary position embeddings, and trains all modalities end‑to‑end through alternating‑modality batches. Omni‑Encoder achieves competitive performance on 16 benchmarks across all four modalities, outperforming individually trained counterparts, and the authors plan to release the code and models publicly.

By Xiaofei Sun, Peng Wang, Huayao Liu, Xiaoxiao Xu, Jun Song, Deqing Li, Lin Qu
arXiv AI
1d ago

LLM-based NPCs in Video Games: A Systematic Mapping Study

The study reviews 377 academic papers on large language model (LLM)–based non‑player characters (NPCs) in video games, focusing on 42 key works. It quantitatively maps component adoption, noting that short‑term memory and world‑state grounding are common (60% and 69%), while long‑term memory, reflection, and retrieval‑augmented generation are less frequent. The paper also reports latency ranges (2.6–15 s) and identifies six structural barriers to commercial adoption, offering evidence‑based recommendations for developers and researchers.

By Dilan Mian, Jeremy Bradbury, Yann-Ga\"el Gu\'eh\'eneuc, Fabio Petrillo, Cristiano Politowski
arXiv AI
1d ago

Constraint-Preserving Flow Matching for Spectral Graph Generation

The paper introduces Laplacian-Constrained Stiefel Flow Matching (LaSFM), a method for generating graphs via their Laplacian eigenvalues and eigenvectors while preserving key structural constraints such as eigenvector orthonormality, positive eigenvalues, and the Laplacian nullspace condition. LaSFM formulates flow matching on a constrained Stiefel manifold and extends to property-guided generation, steering samples toward desired graph properties without violating constraints. Experiments show that LaSFM improves both spectral and graph-level metrics and better aligns generated graphs with target properties.

By Anita Yang, Ken-ichi Kawarabayashi, Yohan Jung
arXiv AI
1d ago

How Finely Can Vision-Language Models See Biology? Bio-VQA: A Benchmark for Taxonomic, Morphological, Spatial and Individual Reasoning

Bio‑VQA is a new, large-scale visual question answering benchmark for biology, comprising 1.86 million questions over 217 k images from 11 datasets. It tests four key aspects of biological visual understanding—taxonomic reasoning, attribute grounding, spatial reasoning, and individual re‑identification—using a graded scoring system that rewards near‑misses more than distant ones. Evaluation of 18 open‑weight vision‑language models shows that general visual capabilities do not transfer well to biological tasks, with markedly lower accuracy on counting, taxonomy, trait measurement, and re‑identification compared to everyday object benchmarks.

By S M Rayeed, Qian Ma, Charles V. Stewart, Yao Ma
arXiv AI
1d ago

Do Saliency Maps Fail Dense Breasts? A Chance-Normalized, Confounder-Aware Audit of Explanation Fairness in Mammography

The study evaluates whether saliency maps—visual explanations for mammography classifiers—perform differently on dense versus non‑dense breasts. Using 1,345 malignant CBIS‑DDSM images, the authors compared Grad‑CAM, Integrated Gradients, and occlusion maps, finding that Grad‑CAM peaks correctly identified 44% of dense‑breast lesions versus 58% of non‑dense ones. After adjusting for case mix (notably more calcifications in dense breasts), the performance gap narrowed, but occlusion maps still showed a significant deficit, and classifiers overall relied less on dense‑breast lesions than on an image‑only conspicuity baseline. The authors recommend reporting density‑specific performance gaps before and after adjustment, along with controls for conspicuity and lesion reliance.

By Ahmed Jaheen
arXiv AI
1d ago

Clips2Story: Training-Free Video Storyboarding and Editing Using Multimodal Retrieval-Embedded Generation

Clips2Story is a training‑free, LLM‑based pipeline that lets users generate short videos from a pool of clips by specifying a keyword. It first extracts multimodal descriptors from each clip, retrieves the most relevant ones, and then uses an LLM to select, arrange, and narrate them into an editable storyboard. The final video is assembled by combining the chosen clips with visuals that match the narration. Evaluation on a synthetic dataset of shuffled clips from 100 long‑form videos shows that Clips2Story achieves better narrative coherence, keyword alignment, and realism than a strong baseline.

By Junnuo Wang, Weihan Xu, Hao-Wen Dong
arXiv AI
1d ago

FedFiOT: Fisher-Weighted Optimal Transport for Model Merging in One-Shot and Few-Round Federated Learning

FedFiOT introduces a Fisher-weighted optimal transport framework for merging client models in one-shot and few-round federated learning. By using Fisher information to assess parameter importance, it aligns mismatched neurons and weights parameters during aggregation. The method alternates between optimizing neuron correspondences and global model parameters, and theoretical analysis shows monotonic improvement of the merging objective with a bound on heterogeneity effects. Experiments on multiple datasets confirm consistent performance gains over existing baselines.

By Quan Minh Nguyen, Hoang M. Ngo, Bhakti Chowkwale, Yasaman Keshtkarjahromi, My T. Thai
arXiv AI
1d ago

UpSafe$^\circ$C: Upcycling for Controllable Safety in Large Language Models

UpSafe°C is a unified framework that enhances large language model safety by upcycling safety‑critical layers into a sparse Mixture‑of‑Experts structure, with a router acting as a soft guardrail. It employs a two‑stage supervised fine‑tuning strategy to improve safety discrimination while preserving general capabilities, and introduces a safety temperature mechanism for dynamic inference‑time control of the safety‑utility trade‑off. Experiments across multiple benchmarks and model scales show robust safety improvements against harmful and jailbreak inputs while maintaining competitive general performance.

By Yuhao Sun, Zhuoer Xu, Shiwen Cui, Kun Yang, Lingyun Yu, Yongdong Zhang, Hongtao Xie
arXiv AI
1d ago

From Outcome to Representation: Tracing Reasoning Mechanisms through Integrated Policy Gradient

The paper introduces Integrated Policy Gradient (IPG), a training‑free framework that applies policy‑based attribution to large language model reasoning. IPG backpropagates outcome‑level, non‑differentiable signals through entire inference trajectories, using path integration to isolate outcome‑relevant mediators at the component level. Empirical results show that IPG precisely localizes sparse internal components linked to specific reasoning behaviors and enables modulation of reasoning capabilities.

By Changming Li, Kaixing Zhang, Haoyun Xu, Yingdong Shi, Zheng Zhang, Kaitao Song, Kan Ren
arXiv AI
1d ago

Certified Circuits: Stability Guarantees for Mechanistic Circuits

Certified Circuits introduces a framework that wraps any black‑box circuit discovery algorithm with randomized data subsampling to provide provable stability guarantees. By certifying that inclusion decisions over circuit components are invariant to bounded edit‑distance perturbations of the concept dataset, unstable components are abstained from, producing more compact and accurate mechanistic explanations. Across ResNet, ViT, and GPT‑2 on vision and language tasks, Certified Circuits achieves up to 56% higher accuracy and up to 80% fewer components, maintaining reliability where baseline methods degrade.

By Alaa Anani, Tobias Lorenz, Bernt Schiele, Mario Fritz, Jonas Fischer
arXiv AI
1d ago

A Roadmap to Impactful Pluralistic Alignment Research

The paper outlines a roadmap for advancing pluralistic value alignment in AI, noting that current research lacks evidence of influence on model training or evaluation. It identifies three key barriers: a need for empirical studies on benefits and drawbacks, clearer specifications of when and how pluralistic behavior should occur, and robust evaluation methods suitable for production. The roadmap aims to guide research toward practical deployment in responsible AI systems.

By Elinor Poole-Dayan, Jillian Fisher, Jacob Andreas, Mitchell Gordon, Michiel A. Bakker
arXiv AI
1d ago

Reliability Engineering for AI Systems: Challenges, Methods, and Directions

The paper "Reliability Engineering for AI Systems: Challenges, Methods, and Directions" discusses how AI reliability extends beyond correct outputs to include consistent operation of retrieval, memory, tool use, permissions, human oversight, and inter-system interactions, especially for generative AI. It adapts traditional reliability engineering techniques—such as failure definitions, operational envelopes, FMEA, accelerated testing, field monitoring, and reliability growth—to AI, proposing a four-level diagnostic framework and a TEVV (Test, Evaluation, Verification, Validation) approach. The authors also introduce the SMART framework for statistical guidance and reference NIST’s AI risk management framework, illustrating their concepts with three case studies and emphasizing the urgent need for new measurements and safety guardrails as AI systems evolve rapidly.

By Rong Pan, Yili Hong, Min Xie
arXiv AI
1d ago

MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability

MASBench is a benchmark for evaluating large language model (LLM)-based multi‑agent systems (MAS) under partial observability, a realistic constraint where agents only see limited environmental information. The benchmark is structured into three progressive task categories—Reasoning, Scheduling, and Game—and assesses three collaboration mechanisms: Protocol, Memory, and Routing. It provides deterministic evaluation metrics such as performance score, communication cost, and cost effectiveness, and includes empirical results across various LLM backbones to guide effective MAS design.

By Qizhi Chu, Zekai Yu, Sijie Wen, Yang Liu, Chen Qian, Cheng Yang, Chuan Shi, Zhiyuan Liu
arXiv AI
1d ago

VCoT-SQL: Enhancing Text-to-SQL Generation through Dual Chain-of-Thought Validation Training

VCoT‑SQL is a framework that improves text‑to‑SQL generation by training two complementary models on curated reasoning traces. It uses clause‑level chain‑of‑thought templates verified through execution and schema alignment, and a validator that detects syntactic, semantic, and type errors. The approach achieves high execution accuracy on BIRD and Spider datasets, outperforming larger closed‑source models while relying only on 7B open‑source models.

By Rui Zhang, Yujia Su, Ruixuan Li, Yuhua Li, Yixin Su
arXiv AI
1d ago

AgenticFinBench: Benchmarking Financial Agents on Diverse Tasks from Real-World Requests

AgenticFinBench is a new benchmark comprising 230 real‑world financial tasks across eight categories, two languages, and four complexity levels, designed to capture the breadth and open‑ended nature of financial AI work. It introduces a reference‑free evaluation framework that uses question‑specific rubrics and a two‑stage claim audit to detect hallucinations, aligning assessment more closely with real‑world financial practice. Experiments on 18 agent configurations show that ignoring hallucinations inflates performance, that harness differences drive evidence grounding more than model differences, and that task complexity reduces quality scores, highlighting challenges in explaining financial impacts and comparing data consistently.

By Shu Zhou, Ziyang He, Yuxin Rao, Yudong Fang, Jiali Qian, Dongyuan Chen, Juntao Zhang, Yong Hu