AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Machine Learning
Oct 9

A Unified Bellman Operator for Safety-Critical Reinforcement Learning

The paper introduces a new Bellman operator that merges performance and safety objectives into a single joint value function for reinforcement learning. Temporal difference learning with this operator converges under a two‑timescale stochastic approximation, estimating safety on a fast timescale and joint value on a slow one. Theoretical analysis shows convergence to safety‑constrained optimal policies, and experiments on continuous control tasks confirm stable learning with near‑zero safety violations.

By Nishanth Arun Rao, Royina Karegoudra Jayanth, Benjamin Eysenbach, Jaime Fern\'andez Fisac
arXiv AI
Oct 9

Speedbumps: Rejection Attacks on Speculative Decoding

Speedbumps: Rejection Attacks on Speculative Decoding explores how Speculative Rejection Attacks (SRAs) can degrade the efficiency of speculative decoding by causing draft and target models to disagree more often, leading to more target-model forward passes per token. The authors introduce two attacks—Speedbump-P and Speedbump-D—that append adversarial suffixes to attacker-controlled content, optimizing the expected length of the accepted speculative prefix. In some scenarios, these attacks slow inference below autoregressive decoding, reduce output quality, yet remain effective under sampling and transfer across drafters or target models sharing a drafter.

By Adam Y. J. Jones, Yu Yuan, Sergio Maffeis
arXiv Computer Vision
Oct 9

When Scene Text Hijacks the Scene: Uncovering, Exploiting, and Mitigating Rendered-Text Semantic Leakage in Image Generation Models

The paper investigates how rendered text in image generation models can leak semantic information, acting as an unintended control channel. By separating the visual prompt from the text, the authors measure how text semantics influence generated images and show that harmful meanings can persist even when the main prompt is benign. They also propose a mitigation strategy that limits unsafe semantic transfer while maintaining correct text rendering.

By Feifei Li, Runjie Wang, Xiaohan Zhang, Zhenxing Qian, Mi Wen, Mi Zhang
arXiv Computer Vision
Oct 9

PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology

PathLang is a new, language‑centered benchmark for vision‑language models in computational pathology. It keeps the underlying pathology slides, ground‑truth labels, and image‑text evaluation direction constant while systematically varying the diagnostic language used in prompts, reflecting how pathologists actually rephrase diagnoses. The benchmark covers four task families—zero‑shot classification with image‑text alignment, cross‑modal retrieval, paraphrase robustness, and open‑vocabulary diagnosis retrieval—and evaluates nine VLMs across five public datasets spanning four organs.

By Fanqi Cheng, Kuo Gong, Shangke Liu, Beidi Zhao, Junchao Zhu, Zheyu Zhu, Leiyue Zhao, Fengbei Liu, John Cannon, Gang Wang, Zu-hua Gao, Kenji Ikemura, Yihe Yang, Yaohong Wang, Yuankai Huo, Xiaoxiao Li, Mert R. Sabuncu, Ruining Deng
arXiv Computer Vision
Oct 9

Perceptually Grounded and Semantics-Aware Evaluation for Holistic Co-Speech Gesture Generation

The paper introduces a new benchmark for evaluating holistic, semantics-aware co‑speech gesture generation, combining 13 objective metrics and a novel Semantic Gesture Preservation (SGP) metric that measures how well ground‑truth semantic gestures are retained. A perceptual study with 101 participants rates generated gestures on five dimensions, revealing that SGP aligns with speech‑aware human judgments while other metrics do not. Composite metrics tailored to each dimension further improve alignment, especially for animation errors and content match, underscoring the need to validate objective metrics against human perception.

By Nick Milkin, Lanmiao Liu, Esam Ghaleb, Asli Ozyurek, Zerrin Yumak
arXiv Computer Vision
Oct 9

GenIA: Generative Reconstruction with Test-Time Input Alignment

GenIA is a framework that aligns generative 3D foundation models with test‑time observations, improving pose estimation and reconstruction from monocular, multi‑view, and dynamic inputs. It derives translation and scale from geometry, retains learned rotation priors, and aligns appearance using visibility‑biased attention, cross‑observation fusion, and differentiable rendering guidance during denoising. An optional post‑denoising refinement further adapts appearance latents and object placement, and the method also supports externally supplied geometry for dynamic objects, achieving better results than recent optimization‑based and image‑to‑3D methods on synthetic and real benchmarks.

By Stefano Esposito, Naama Pearl, Polina Karpikova, Samuel Rota Bul\`o, Lorenzo Porzi, Peter Kontschieder, Andreas Geiger, Jonathon Luiten
arXiv Computer Vision
Oct 9

False Claims, Credible Images: A Red-Teaming Benchmark for Commercial Image Generators

The paper introduces EpiReal-Bench, a benchmark of 10,000 false-claim prompts and corresponding images designed to test visual misinformation risks in commercial image generators. It also presents EpiReal-Attack, a black-box optimization framework that can bypass alignment safeguards, increasing the success rate of generating misleading images from 70% to 95%. Experiments on four commercial models show that these generators can produce credible visual evidence for false claims, highlighting a blind spot in current alignment methods.

By Zeyu Ye, Yanchun Li, Sibei He, Meng Xie, Hangtao Zhang, Xianlong Wang, Li Zeng, Jiahao Chen, Yichen Wang, Junhui Wang, Ziqi Zhou
arXiv Computer Vision
Oct 9

DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training

DreamTrue is a multi‑view, cross‑embodiment robot world model that predicts physically plausible videos while faithfully following actions. It overcomes calibration and limited interaction coverage issues by rendering action trajectories into image‑space conditions, applying offline geometric calibration, and using counterfactual post‑training to generate a broader range of future videos. A human‑annotated defect dataset trains a video reward model that guides reinforcement‑learning post‑training, resulting in state‑of‑the‑art action following on AgiBot and a significant drop in interaction defect rates.

By Junyan Li, Ruizhi Li, Yu Liu, Xiangshuo Liu, Mingchao Sun, Hongyu Pan, Mu Xu, Lue Fan, Zhaoxiang Zhang
arXiv Computer Vision
Oct 9

DDL: A Large-Scale Dataset for Deepfake Detection and Localization in Diversified Real-World Scenarios

The paper introduces DDL, a large-scale dataset for deepfake detection and localization, comprising over 1.4 million forged samples generated by 80 distinct deepfake methods. It features comprehensive coverage of generation architectures, varied manipulation modes, diverse scenarios and modalities, and provides fine-grained spatial masks and temporal segments for each forgery. These enhancements aim to offer a more challenging benchmark and support the development of next-generation deepfake detection, localization, and interpretability methods.

By Changtao Miao, Yi Zhang, Weize Gao, Zhiya Tan, Weiwei Feng, Man Luo, Jianshu Li, Ajian Liu, Yunfeng Diao, Qi Chu, Tao Gong, Zhe Li, Weibin Yao, Joey Tianyi Zhou
arXiv Computer Vision
Oct 9

BiomedXPro: Prompt Optimization for Explainable Diagnosis with Biomedical Vision Language Models

BiomedXPro is an evolutionary framework that uses a large language model to automatically generate a diverse set of interpretable, natural‑language prompt pairs for disease diagnosis with biomedical vision‑language models. It addresses the lack of transparency in existing prompt optimization methods by producing prompts that are both understandable and aligned with clinically significant features. Experiments on multiple biomedical benchmarks show that BiomedXPro outperforms state‑of‑the‑art prompt‑tuning techniques, especially in few‑shot, data‑scarce scenarios.

By Kaushitha Silva, Mansitha Eashwara, Sanduni Ubayasiri, Ruwan Tennakoon, Damayanthi Herath
arXiv Computer Vision
Oct 9

Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts

The paper argues that multimodal large language models can answer vision benchmarks using only linguistic cues, revealing non-visual shortcuts. It proposes a Test‑set Stress‑Test (TsT) that trains a text‑only model on the test set to expose exploitable patterns, and introduces Iterative Bias Pruning (IBP) to remove high‑bias questions. Applying these methods to VSI‑Bench, CV‑Bench, and MMMU shows significant gaps between blind and vision‑aware performance and demonstrates that debiased benchmarks can reduce blind scores while preserving vision scores.

By Ellis Brown, Jihan Yang, Shusheng Yang, Rob Fergus, Saining Xie
arXiv Computer Vision
Oct 9

Super-Resolution in The Right Latent Space: A Frozen Vision-Foundation Substrate

The paper proposes that super‑resolution should be performed in a latent space that is close to the manifold of high‑resolution images, rather than in reconstruction‑oriented or pixel spaces. By using the latent space of 23 fused layers of a frozen DINOv3‑L model, degraded images are embedded near this manifold, enabling a 415M decoder (RAESR) to map them back and decode to pixels in a single pass. RAESR achieves the best fidelity–perception trade‑off on several benchmarks while running at 37 ms per 512×512 image on a single H20 GPU, outperforming models that use a VAE latent substrate.

By Wanzhou Lei, Cuifeng Shen, Yanjin He, Maohua Li, Hua Yuan, Per-Olof Persson, Tao Lan, Kan Liu, Hanlin Tang
arXiv Statistics ML
Oct 9

Deep reinforcement learning for optimal trading with partial information

The paper explores optimal trading under partial information, modeling the trading signal as an Ornstein‑Uhlenbeck process with regime switching. It proposes three reinforcement‑learning approaches that integrate recurrent neural networks to filter latent market regimes, comparing a one‑step method and two two‑step methods that use posterior regime probabilities or signal forecasts. Simulations and an equity pairs‑trading case study show that using posterior probability estimates of the latent regime yields the best cumulative rewards and more interpretable strategies.

By Andrea Macr\`i, Sebastian Jaimungal, Fabrizio Lillo
arXiv Machine Learning
Oct 9

HAND: A Biologically-Inspired Activation Function that Improves Generalisation and Sample Efficiency in Image Classification

The paper introduces HAND, a biologically-inspired activation function that incorporates homeostasis, accelerating nonlinearity, and divisive normalization to act as an inductive bias in deep neural networks. Experiments on image classification show that using HAND allows a ConvNeXt-tiny model to reach ImageNet1k accuracy in 25 epochs versus 200 epochs for the baseline, and yields larger accuracy gains on long-tailed and reduced-data settings. The authors report that HAND does not degrade generalisation on common corruptions and can improve the model’s ability to reject unknown classes, with benefits observed across multiple CNN architectures and datasets.

By Michael W. Spratling, Heiko H. Sch\"utt
arXiv AI
Oct 9

Distillation for Incrimination and Distillation for Capabilities

The paper introduces two distillation methods for handling misaligned AI models. Distillation for Incrimination (DFI) transfers misalignment without the ability to conceal it, making students more likely to admit hidden behaviors. Distillation for Capabilities (DFC) transfers useful capabilities while minimizing misalignment, using techniques like inoculation prompting and focused training.

By Sebastian Prasanna, Jacqueline Tay, Alek Westover
arXiv AI
Oct 9

When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection

The paper introduces PARCEL, a benchmark designed to evaluate whether legal claims are supported by cited authorities. Using 3,396 claims from New York State Court of Appeals decisions, the authors label each claim as Supported, Refuted, or Not Found and cast the task as a three‑way natural language inference problem. State‑of‑the‑art LLMs achieve up to 0.97 accuracy in zero‑shot settings, yet they still frequently misclassify unsupported claims as supported, especially when fabricated but plausible citations are present.

By M. Mikail Demir, M. Abdullah Canbaz
arXiv Machine Learning
Oct 9

When Listening Becomes Easier: Scrubbing Visual Cues for Shortcut-Free VLAs

The paper addresses shortcut learning in vision‑language‑action (VLA) models, showing that different backbones vary in their susceptibility to visual shortcuts that arise from limited demonstration diversity. It introduces a representation‑level metric called action margin to predict shortcut use without rollouts, and proposes task scrubbing—a domain‑adversarial training method—to reduce reliance on visual cues and enhance out‑of‑distribution robustness. Experiments in simulation and real‑world settings demonstrate that task scrubbing improves generalization and often eliminates visual shortcut learning.

By Jasper Gerigk, Kenzo Aspuru-Takata, Chin-Hsuan Wu, Mohammad Mohammadi, Shuhong Zheng, Igor Gilitschenski
arXiv Machine Learning
Oct 9

Higher-Order Morphology Priors for Quadruped Reinforcement Learning Under Actuator Degradation

The paper investigates how incorporating higher‑order mechanical structure into reinforcement learning policies can improve quadruped locomotion when actuators degrade. By modeling the Unitree Go1 robot as a cell complex and using Hodge‑based message passing across node‑edge‑face relationships, the authors show that this higher‑order morphology actor outperforms other approaches on unseen actuator degradations, achieving higher returns, better survival, and lower velocity‑tracking error. The study demonstrates that explicitly encoding complex mechanical relationships provides a powerful inductive bias for whole‑body compensation in degraded systems.

By Derek You, Zafir Shamsi, Keqin Wang, Christine Allen-Blanchette
arXiv AI
Oct 9

BridgeGuard: Explicit Safety Drift for Diffusion-based Autonomous Driving

BridgeGuard is a safety‑constrained diffusion planning method for autonomous driving that progressively strengthens a constraint term during denoising, guiding intermediate trajectories toward a scene‑dependent safety domain. It operates in a low‑dimensional curve space and uses a learned DistanceFieldNet to predict a time‑dependent distance field from bird’s‑eye‑view features, providing safety injection while keeping the perception backbone and planner frozen. Experiments on Bench2Drive show that BridgeGuard improves driving scores and success rates across two diffusion‑based models, demonstrating cross‑model generalization.

By Zhenjun Qiu, Jianing Huang, Dongang Liu, Baiyu Du, Yixun Niu, Hao Yang, Xinyu Huang, Chuan Hu, Shu Liu
arXiv Machine Learning
Oct 9

Uncertainty-Aware Optimization for Physics-Aware Highway Trajectory Prediction

The paper introduces uncertainty-aware extensions of the X-TRACK physics‑aware trajectory prediction framework, namely X-TRACK-DE and X-TRACK-MCD. These extensions predict future vehicle motion variables while explicitly modeling and propagating both aleatoric and epistemic uncertainties through vehicle dynamics, rather than only in trajectory space. Conformal prediction is then applied to the trajectory‑space covariance to create uncertainty regions that achieve a desired marginal coverage level, and experiments on the highD dataset demonstrate improved prediction accuracy and calibrated uncertainty estimates.

By Aanchal Rajesh Chugh, Sebastian Dorn