AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Machine Learning
Oct 28

Dream to Assist: Intent-Conditioned World Models for Shared Vehicle Control

Dream2Assist is a model‑based reinforcement learning framework that extends a recurrent state‑space world model with intent inference to support shared vehicle control. The system is trained via fictitious co‑play to anticipate discrete human objectives without overriding agency, and includes a full deployment pipeline to translate the policy into a safe, human‑centered real‑world system. Validation through three human‑subject studies and on‑vehicle deployments in high‑speed racing and urban driving shows that intent conditioning improves torque alignment and reduces collision rates.

By Jonathan DeCastro, Thomas Balch, Emily Sumner, Deepak Gopinath, Laporsha Dees, Patricio Reyes Gomez, Elliot Weiss, Megan Applegate-Kenton, Todd Rowell, John Leonard, Avinash Balachandran, Guy Rosman
arXiv Machine Learning
Oct 28

Distributed Predictive Flocking for a Swarm of Miniature Vibration-Driven Robots in a Confined Arena

The paper introduces a distributed predictive flocking controller for a swarm of miniature vibration‑driven robots operating in a confined arena without a leader or global map. Each robot solves a nonlinear model predictive control problem in real time, optimizing cohesion, alignment, safety, and task objectives while exchanging only predicted trajectories with neighbors. Experiments on a digital twin and real robots show that a look‑ahead discrete‑time control‑barrier‑function formulation yields the fewest collisions and widest safety margins across various swarm sizes, maintaining coordination and cohesion even near the geometric packing limit.

By Prajol Shrestha, Siddhant Mohapatra, Filip Novkoski, Julian Oelhaf, Nicolas Vandewalle, Ana-Sun\v{c}ana Smith, Andreas Maier
arXiv Machine Learning
Oct 28

Security of Deep Reinforcement Learning for Autonomous Driving: A Survey

The paper surveys 86 recent studies on the security of reinforcement learning (RL) in autonomous driving. It systematically categorizes attacks and defenses by threat models and single- versus multi-agent settings, addressing gaps in previous surveys. The authors also evaluate how current attacks and defenses apply to autonomous driving, offering guidance for building robust RL systems.

By Ambra Demontis, Srishti Gupta, Maura Pintor, Luca Demetrio, Kathrin Grosse, Hsiao-Ying Lin, Chengfang Fang, Battista Biggio, Fabio Roli
arXiv Machine Learning
Oct 28

Position: Adopt Constraints Over Fixed Penalties in Deep Learning

The paper argues against using fixed weighted-sum penalties to enforce constraints in deep learning, citing three main issues: non-equivalence of penalized and constrained problems in non-convex settings, difficulty in selecting appropriate penalty coefficients, and the softening of hard requirements. It contends that these problems can lead to non-compliance and relaxed constraints. The authors recommend directly solving the constrained problem when non-negotiable requirements are present.

By Juan Ramirez, Meraj Hashemizadeh, Simon Lacoste-Julien
arXiv Machine Learning
Oct 28

Online Learning in Stabilized Linear Dynamical Games with Adversarial Disturbances

The paper investigates online learning for multiple strategic agents interacting within a shared linear dynamical system that experiences adversarial disturbances. Each agent aims to minimize its own convex loss while observing the full state, and the study extends the stabilizing-baseline approach from nonstochastic online control to this multi‑agent setting. By employing online gradient descent with memory under state‑only and aggregate‑input feedback, the authors establish sublinear, near‑optimal per‑agent regret bounds and, in the common‑interest case, demonstrate equilibrium‑tracking guarantees via a time‑varying potential game framework.

By Anas Barakat, John Lazarsfeld, Georgios Piliouras, Antonios Varvitsiotis
arXiv Machine Learning
Oct 28

From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme

The paper introduces HUMOR, a framework that enhances vision‑language models (VLMs) for generating humorous memes by guiding them through a hierarchical, multi‑path Chain‑of‑Thought reasoning process. HUMOR first identifies a template‑level intent, explores diverse reasoning paths under different contexts, and then anchors onto a high‑quality, context‑specific path, thereby increasing reasoning diversity. Additionally, a pairwise reward model trained on groups of memes sharing the same template captures subjective humor, enabling group‑wise reinforcement learning that guarantees monotonic improvement within a trust region. Experiments demonstrate that HUMOR improves reasoning diversity, preference alignment, and overall meme quality, and the authors suggest the approach generalizes to other open‑ended, human‑aligned multimodal generation tasks.

By Xueyan Li, Yingyi Xue, Mengjie Jiang, Qingzi Zhu, Yazhe Niu
arXiv Machine Learning
Oct 28

Probabilistic adjustment of seasonal forecasts using cVAE with a non-parametric decoder: A case study of Arctic sea ice predictions

The paper presents a probabilistic post‑processing framework that uses a conditional Variational Autoencoder (cVAE) with a non‑parametric decoder to adjust seasonal Arctic sea‑ice forecasts. By replacing the Gaussian decoder with a generator and optimizing with Continuous Ranked Probability Score (CRPS), the method produces larger, bias‑corrected ensembles that are better calibrated, more consistent with observations, and exhibit higher resolution and sharper spectral power than standard cVAEs. The approach demonstrates improved forecast skill and reduced errors compared to benchmark predictions.

By Parsa Gooya, Reinel Sospedra-Alfonso
arXiv Computation and Language
Oct 28

No Tokenizer, Still a Gap: Cross-Lingual Fairness of Homoglyph and Invisible-Character Watermarking

The paper investigates whether edit‑based watermarking techniques—homoglyph substitution and invisible‑character insertion—avoid the cross‑lingual fairness gaps seen with generation‑time watermarks that depend on tokenizers. Across 40 languages in 24 scripts, the authors measure watermark capacity per character, its resilience to routine processing and targeted removal, and the detector effort required. Results show that script choice explains most of the capacity variation, with lower‑resourced languages using scripts that offer fewer homoglyphs, and that ordinary copy‑paste operations further widen the gap by stripping invisible characters.

By Alexander Nemecek, Yuqiao Xu, Joseph Falkenburg, Erman Ayday
arXiv Computation and Language
Oct 28

Decision Models as Judges for Hallucination Detection

Decision models, which provide predefined verdicts with associated probabilities, are evaluated as a low‑cost alternative to large language models used as judges for hallucination detection. In a benchmark of 9,852 items across five hallucination types, the best decision model matches the generative judge GPT‑6‑sol on context inconsistency and is within 3 percentage points on factual errors, though it lags on logical and instruction inconsistency. A confidence cascade of decision models improves performance by 1.9–3.3 points on three types and can match or exceed GPT‑6‑sol at only 1–3% of its cost.

By Haozheng Liu, Zhaoqian Xue, Zhongyin Wang, Junting Mu, Zihui Ma, Shuya Feng, Lingyao Li
arXiv Computation and Language
Oct 28

Same Verdict, Different Readers: Personalizing Fact-Checking Explanations from Relative Preferences

The paper introduces PERPL, a framework for tailoring fact‑checking explanations to individual readers while preserving factual accuracy. It learns personalized rewriting from relative preferences of simulated LLM readers, using cross‑persona pairing to generate supervision and a progressive learning strategy to refine explanations beyond predefined styles. Experiments show PERPL outperforms larger general‑purpose LLMs in aligning with reader preferences while maintaining high factual fidelity.

By Haokun Wen, Xuemeng Song, Mong-Li Lee, Wynne Hsu
arXiv Computation and Language
Oct 28

Teach reality, not ethics: a drop-in continued-pretraining corpus installs coupled human-biosphere-AI welfare dispositions across model families

The paper introduces a continued‑pretraining corpus called "coupled welfare" that embeds ecological reasoning into large language models. After a single QLoRA run, the corpus was applied to five open models across three architectures, resulting in a dramatic reduction of harmful decisions under high‑pressure scenarios while leaving general performance unchanged. The study also demonstrates that the improvement is content‑specific, as a neutral control corpus did not yield similar gains, and that the models remain robust against adversarial attempts to justify environmentally destructive actions.

By Trent R. Northen, Mingxun Wang
arXiv Computation and Language
Oct 28

Multi-State Linear Attention with Generalized Second-Order Routing

Multi-State Linear Attention (MSLA) improves long‑sequence modeling by compressing context into separate memory states, but its aggregation only uses mean alignment, ignoring distribution within segments. MERLIN (MomEnt Routing for multi‑state LINear attention) enhances this by tracking first and second moments of each segment’s keys, enabling queries to recover both mean and variance of alignment for more content‑aware aggregation. Experiments on Gated DeltaNet‑1.3B and Mamba‑2‑780M demonstrate that MERLIN consistently outperforms existing aggregation methods on long‑context tasks, achieving significant accuracy gains while using less memory.

By Hee Seung Hwang, Sanghyuk Chun, Xindi Wu, Olga Russakovsky, Zhiwei Deng
arXiv Computation and Language
Oct 28

Mining the Nineteenth Century for the Thirteenth: Register-Scale Forced Alignment and HTR Bootstrapping via Legacy Regesta and Antiquarian Editions

The study evaluates how nineteenth‑century scholarship—specifically Pressutti’s regesta and partial editions—can aid handwritten text recognition (HTR) of Pope Honorius III’s registers beyond the limited modern expert transcriptions available for regnal years 1‑2. By aligning legacy layers with modern source lines, the authors demonstrate that projection onto curial spelling with replay reduces word‑error rate by 1.50 points, that a legacy lexicon covers 82.2 % of out‑of‑vocabulary words and flags errors with 66.7 % precision, and that a distilled medieval NER model achieves an F1 of 82.1 % on a validated benchmark. Additionally, Pressutti’s regesta and related topological data recover 73.9 % of entry starts and suggest a character‑error rate of 7.80 % beyond folio 203, indicating that antiquarian scholarship can steer, check, and segment HTR rather than replace modern transcription.

By Tamas Kovacs, Daniel Luger
arXiv Machine Learning
Oct 28

Cumulative Marginal Matching for One-Step Diffusion

The paper introduces Cumulative Marginal Matching (CMM), a new objective for training few‑step diffusion and flow‑matching models. CMM treats flow‑matching as a sequence of intermediate marginal distributions and directly optimizes the cumulative Wasserstein‑2 discrepancy between the base and few‑step models across the entire trajectory. It derives a tractable surrogate for this objective, shows that Shortcut‑CSL is a first‑order approximation of CMM, and reports improved generation quality on standard benchmarks under the same training budget.

By Paribesh Regmi, Rui Li
arXiv Computation and Language
Oct 28

BanglaBBQ: A Bias Benchmark for Question Answering in the Bangladeshi Context

BanglaBBQ is a question‑answering bias benchmark that captures Bangladeshi stereotypes in both Bangla and English. It adapts the original BBQ by classifying templates into directly translated, modified for Bangladesh, and removed categories, retaining 140 templates across six groups. Five new stereotype categories—marital status, age and political perception, regional origin, educational background, and mental‑health status—were added, with 466 new templates evaluated and 372 retained after annotation. The dataset was tested on Qwen3‑4B, Qwen3‑8B, GPT‑4.1‑mini, and Gemini 2.5 Flash, revealing that all models tend to favor stereotyped individuals when context is ambiguous, with language‑specific differences and varying accuracy depending on stereotype confirmation or contradiction.

By Sumit Anthony Gomes, Yasin Rahman, Taeeba Tasnia, Faiza Nooren Suhita
arXiv Computation and Language
Oct 28

Decoupling and Regulating Modality Heads for Fake News Detection under Missing Modality

The paper introduces Head-wise Modality Decoupling and Regulation (HMDR) to improve fake news detection when some modalities (image or text) are missing. HMDR first decouples modality representations via soft head grouping based on attention preference, then regulates head contributions to balance modality attention, and finally enforces modality invariance by aligning representations when one modality is replaced. Experiments on two benchmarks demonstrate that HMDR enhances detection performance under missing modalities while preserving accuracy on complete inputs.

By Kai Qian, Dongjie Xu, Weijie Shi, Yezhao Xu, Jiaqi Wang, Hanghui Guo, Mengze Li, Hao Chen, Ziyi Liu, Jiajie Xu
arXiv Computation and Language
Oct 28

Who Gets Flagged? The Pluralistic Evaluation Gap in AI Content Watermarking

The paper "Who Gets Flagged? The Pluralistic Evaluation Gap in AI Content Watermarking" discusses how AI content watermarking, used for authentication and provenance, varies in signal strength and robustness across different languages, cultural visual traditions, and demographic groups. It reviews existing benchmarks and finds that most do not report performance across these dimensions, highlighting a bias risk. The authors propose three evaluation dimensions—cross‑lingual detection parity, culturally diverse content coverage, and demographic disaggregation of detection metrics—to ensure fair and inclusive watermarking practices.

By Alexander Nemecek, Osama Zafar, Yuqiao Xu, Wenbiao Li, Erman Ayday
arXiv Computer Vision
Oct 28

Tail Cues, Principal Corrections: Plug-and-Play Rectification for Open-Set Test-Time Adaptation

Tail Cues, Principal Corrections: Plug-and-Play Rectification for Open-Set Test-Time Adaptation introduces TALOR, a lightweight module that rectifies features for non-stationary target streams by decomposing them into high-gain principal and low-gain tail coordinates. TALOR applies a soft-routed affine correction in the principal subspace, subtracts drift via weighted ridge regression, and reconstructs the feature for the source head. On ImageNet-C with six csOOD datasets, TALOR surpasses the next-best UniEnt by 2.2% H-score, runs 2.5× faster, and uses only 12% of its GPU memory, while also boosting COME to 68.4% H-score with minimal memory overhead.

By Yingkai Yang, Chaoqi Chen, Hui Huang
arXiv Computer Vision
Oct 28

Diagnosing and Correcting Bias in MLLM for Long Video Understanding

The paper introduces S‑MME, a benchmark for diagnosing shortcut‑biased behavior in Multimodal Large Language Models (MLLMs) on long‑video question answering. It proposes CLEAR, a training‑free framework that estimates and mitigates bias by treating the full video‑question pair as a complete view and intervening on hidden states during inference. Experiments on five benchmarks demonstrate that CLEAR improves prediction accuracy and robustness across different MLLMs.

By Xusheng Liang, Jianqiao Sun, Hao Zhang, Yulei Niu, Hengshuang Zhao, Jiawei Ma
arXiv Computer Vision
Oct 28

Evaluating Federated Vision-Language and Vision-Only Models for Privacy-Aware Breast Cancer Imaging

The paper evaluates privacy‑aware federated learning for breast cancer imaging by comparing pretrained vision‑language models (CLIP, BiomedCLIP) and a ViT‑B/16 baseline on two public datasets, with and without differential privacy. All five models achieve high normal‑versus‑sick discrimination (ROC‑AUC 0.940‑0.990), but acquisition year alone attains 0.974, indicating potential confounding. Removing the text pathway keeps accuracy while cutting communication cost, while differential privacy lowers ROC‑AUC by 0.375 due to perturbing the full model state.

By Malithi Mithsara Wanniarachchi Kankanamge, Ning Yang, Sree Bash Debnath, Md Zobaer Islam, Abdur R. Shahid