AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Machine Learning
Oct 22

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

The paper introduces a model‑free continuous‑time reinforcement learning algorithm that learns from a fixed batch of discrete‑time off‑policy data. It builds on the PhiBE‑Q framework to create a PhiBE‑PI policy iteration that uses a single state‑action critic, avoiding explicit drift or diffusion estimation. The authors provide a local end‑to‑end suboptimality guarantee under linear function approximation, analyze error propagation from discretization, function approximation, and finite samples, and reveal a bias‑variance trade‑off where finer time discretization can worsen statistical error.

By Chaorui Yao, Yuming Paul Zhang, Lin Yang, Yuhua Zhu
arXiv Computer Vision
Oct 22

AdvFidelity: Unifying Preference Alignment and Few-Step Image Generation within Reinforcement Learning

AdvFidelity unifies preference alignment and few-step image generation by embedding artifact-sensitive fidelity supervision directly into reinforcement learning. It trains a discriminator to rank high-quality references above content-matched reconstructions, enabling online adaptation of fidelity rewards without explicit distillation. The method achieves alignment, task performance, and generation quality comparable to full-step LeapAlign while using only 4 or 8 steps and no classifier-free guidance.

By Sicong Kang, Fangao Zeng, Yuhan Li, Hao Zhou, Mengfei Xu, Wenxiang Shang, Wei Li, Bingbing Ni, Pipei Huang
arXiv Machine Learning
Oct 22

Implicit Feature Reweighting for Label-Free Graph Generalization

SCARCE is a label‑free representation‑learning framework that improves graph‑level out‑of‑distribution generalization. It uses an environment partition of training graphs—derived from metadata or self‑supervised clustering—to warm‑start a variational graph autoencoder and then applies an adversarial confusion mechanism that prevents a discriminator from distinguishing environments while preserving reconstruction fidelity. Theoretical analysis shows that this environment confusion contracts environment‑discriminative directions, effectively reweighting features to reduce reliance on spurious correlations, and experiments on synthetic and DrugOOD datasets demonstrate state‑of‑the‑art OOD performance among label‑free methods.

By Tianjun Yao, Chen Xi, Xinyang Li, Lixian Zhong
arXiv Machine Learning
Oct 22

SU(2)-Equivariant Spinor Message Passing for Molecular Magnetism

The paper introduces eSpinor, an SU(2)-equivariant graph neural network that propagates spinor features for predicting molecular magnetism. By initializing spinor features from eigenvectors of an approximate spin‑orbit coupling Hamiltonian and employing gauge‑covariant message passing with a gauge‑invariant readout, eSpinor achieves more accurate zero‑field‑splitting predictions for cobalt single‑molecule magnets and improves electron‑spin correlation predictions for a dysprosium MRI probe compared to existing methods. The study demonstrates that incorporating SU(2) symmetry as an inductive bias enhances performance on magnetic properties and highlights that static tensor errors can lead to different dynamical errors, suggesting careful model selection for specific observables.

By Erh-Wei Sheng, Beom Seok Kang, Yuchao Lin, Danish Khan, Anima Anandkumar
arXiv Machine Learning
Oct 22

Privacy in Context Is Being Overlooked in the Age of General-Purpose LLMs

The paper argues that as large language models (LLMs) expand, privacy concerns grow, yet current approaches tend to seek a single, universal privacy solution. It emphasizes that privacy is inherently contextual, and warns that a unified framework can give a misleading sense of security and ignore fundamental tensions among desired privacy properties. The authors propose focusing on application‑specific deployment, context‑specific evaluation, and improved privacy literacy as key future directions.

By Prakhar Ganesh, Yash More, Marco Romanelli, Ferdinando Fioretto, Golnoosh Farnadi
arXiv Machine Learning
Oct 22

Differentially Private Sparse Reward Estimation with Preference Feedback

The paper introduces a differentially private framework for sparse reward estimation in reinforcement learning with human feedback (RLHF). It assumes that only a small subset of features influences human preferences and provides theoretical error bounds for both Sample-DP and Label-DP settings under the Bradley-Terry-Luce model. Empirical results show that the proposed methods outperform dense baselines on sentiment generation and safety alignment tasks, achieving a better privacy‑utility trade‑off.

By Meng Ding, Mingxi Lei, Jie Zhang, Jinyan Liu, Di Wang
arXiv Machine Learning
Oct 22

The Alien Space of Science: Sampling Coherent but Cognitively Unavailable Research Directions

arXiv:2603.01092v3 Announce Type: replace-cross Abstract: Scientific discovery is constrained not only by what is true, but by what is cognitively available to researchers. Many directions are cohere...

By Alejandro H. Artiles, Martin Weiss, Levin Brinkmann, Iyad Rahwan, Bernhard Sch\"olkopf, Christopher Pal, Hugo Larochelle, Anirudh Goyal, Nasim Rahaman
arXiv Machine Learning
Oct 22

MCLR: Improving Conditional Modeling via Inter-Class Likelihood-Ratio Maximization and Unifying Classifier-Free Guidance with Alignment Objectives

arXiv:2603.22364v4 Announce Type: replace Abstract: Diffusion models achieve strong generative performance but often rely on classifier-free guidance (CFG), an inference-time intervention that modifi...

By Xiang Li, Yixuan Jia, Xiao Li, Jeffrey A. Fessler, Rongrong Wang, Qing Qu
arXiv Computation and Language
Oct 22

Deep Reasoning in General Purpose Agents via Structured Meta-Cognition

arXiv:2605.11388v2 Announce Type: replace Abstract: Humans intuitively solve complex problems by flexibly shifting among reasoning modes: they plan, execute, revise intermediate goals, resolve ambigu...

By Dean Light, Michael Theologitis, Kshitish Ghate, Shuyue Stella Li, Benjamin Newman, Chirag Shah, Aylin Caliskan, Pang Wei Koh, Dan Suciu, Yulia Tsvetkov
arXiv Computation and Language
Oct 22

AfriBED: a Speech Recognition Bias Evaluation Dataset for Low Resource African Languages

AfriBED is a newly released speech recognition bias evaluation dataset focused on two low‑resource African languages, Bemba and Nyanja. It contains 9 hours of Bemba speech from 80 speakers (3,928 utterances) and about 18 hours of Nyanja speech from 127 speakers (6,260 utterances). Baseline experiments show the data can be used to detect and quantify gender and speaker‑type bias in ASR systems for these languages.

By Claytone Sikasote, Hussein Suleman, Jan Buys
arXiv Machine Learning
Oct 22

Estimating Performance for Safe Deployment of Machine Learning Models

The paper introduces MOVAL, a model evaluation tool that estimates performance of deep neural networks on unlabeled test data by calibrating class-wise confidence scores. MOVAL mitigates issues from distributional shifts and class imbalance, enabling estimation of metrics beyond accuracy such as F1-Score and AUC. Experiments on natural and medical image datasets demonstrate state‑of‑the‑art results across classification and segmentation tasks, providing accurate class, instance, and pixel‑level performance estimates for safe deployment.

By Zeju Li, Konstantinos Kamnitsas, Mobarakol Islam, Melanie Roschewitz, Cheng Ouyang, Chen Chen, Ben Glocker