AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv AI
Sep 7

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

The paper introduces KoNA, a benchmark designed to evaluate selective non‑compliance in vision‑language models (VLMs) across five categories—False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. KoNA tests both query‑level and component‑level non‑compliance using paired single and compound queries, revealing that many VLMs struggle to refuse, correct, or abstain appropriately, especially when selective non‑compliance is required. Fine‑tuning VLMs on KoNA examples improves non‑compliance accuracy while preserving performance on fully answerable tasks, indicating that models can better distinguish answerable components from those needing non‑compliance.

By Minji Kim, Jihyoung Jang, Hyounghun Kim
arXiv AI
Sep 7

Leveraging Imperfect Restoration for Data Availability Attack

The paper introduces Imperfect Restoration Poisoning (IRP), a new data poisoning technique that maintains high image quality while effectively disrupting both supervised and self‑supervised learning models. It builds on a theoretical critique of the existing Convolution-based Unlearnable Dataset (CUDA) method, revealing CUDA’s sub‑optimal gradients and class‑bias strategy. Extensive experiments demonstrate IRP’s superiority over eight baseline attacks and its resilience against five defense methods.

By Yi Huang, Jeremy Styborski, Mingzhi Lyu, Fan Wang, Adams Kong
arXiv Machine Learning
Sep 7

SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Control

SCRIPT is a scalable diffusion policy that uses a Joint Action-State-Text Diffusion Transformer (JAST‑DiT) to jointly encode actions, physical states, and natural‑language instructions, enabling direct interaction between language semantics and control dynamics. The method employs a multi‑stage training framework, including supervised imitation pre‑training, a nonlinear history conditioning mechanism for stable autoregressive control, and a post‑training stage with Reinforcement Learning with Hybrid Rewards (RLHR) that injects learnable noise to improve motion quality and instruction following. Experiments on the 1200‑hour MotionMillion dataset show that SCRIPT outperforms prior state‑of‑the‑art methods across text alignment, motion quality, and physical realism, and its performance scales consistently with model size.

By Jingyan Zhang, Han Liang, Ruichi Zhang, Bin Li, Juze Zhang, Xin Chen, Jingya Wang, Lan Xu, Jingyi Yu
arXiv AI
Sep 7

Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models

The article presents a technical manual for an open toolkit designed to measure how transformer language models individuate word meanings across different contexts. It introduces the concept of a "bridge form"—a single word that appears unchanged in multiple domains but with distinct senses—and outlines a full pipeline from specifying these forms to extracting layer-wise representations, computing silhouette-based separation metrics, and visualizing results. The manual details each design choice and its intended methodological safeguards, emphasizing that it serves as a methodological reference rather than reporting empirical findings.

By Jos\'e Luciano Ver\c{c}osa Marques, Frederico Jorge Heitmann, Daniel Omar Perez, Marcelo Vinicius de Paula, T\'arcio Andr\'e dos Santos Barros
arXiv AI
Sep 7

RegionFed: Federated Learning for Personalized Query Understanding in Heterogeneous Retail Environments

RegionFed is a federated learning framework designed for personalized query understanding in retail search systems that face significant data heterogeneity across geographic regions. By operating at the gradient level and using the β1 conflict between regional and global gradients, it diagnoses heterogeneity, selects appropriate personalization strategies, and controls personalization strength without requiring code changes for different architectures. The method achieves near‑centralized performance (92.27%) on transformers and CNNs while providing differential privacy and theoretical convergence guarantees.

By Quoc H. Nguyen, Ali Lafzi, Abhijeet Phatak, Siddharth Pratap Singh, Rohit Upadhyay, Yogananda Domlur Seetharama, Chittaranjan Tripathy
arXiv AI
Sep 7

One Diffusion Model, Two Roles: Guided Trajectory Planning and Safety-Critical Scenario Generation in Closed-Loop Simulation

The paper presents a single pretrained diffusion traffic model that serves both as an ego motion planner and as a controllable generator of safety‑critical scenarios for autonomous driving. It introduces a Single‑Stream Dual‑Stream diffusion‑transformer decoder (SSDS) that fuses scene context via joint attention, improving closed‑loop performance on the nuPlan benchmark, and a training‑free guidance scheme called Decoupled Annealing Posterior Sampling with Energy (DAPSE) that injects arbitrary energy functions at inference time. Using the same model, the authors generate realistic long‑tail driving interactions—such as aggressive cut‑ins and lead‑vehicle braking—through inference‑time guidance, exposing failure modes in black‑box planners that standard benchmarks miss.

By Arka Pal, Rajesh Kumar, Hannes Eriksson, R\'emi Lacombe, Arvid Laveno Ling, Ankit Gupta, Maciej Wozniak
arXiv Computation and Language
Sep 7

Beneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs

The paper investigates how large language models (LLMs) organize reasoning operations—such as problem formulation, goal decomposition, and deduction—within their hidden representation spaces. It shows that these operations are separable in held‑out representations, with peak separability in middle layers, and that token‑wise alignment of operations becomes more distributed across spans as layers deepen. Attention‑masking experiments reveal that representations aligned to operations at chunk onsets depend on prior reasoning context, indicating a geometric correspondence between linguistic reasoning expressions and internal model structure.

By Seogyeong Jeong, Jaehui Hwang, Dongyoon Han, Geonmo Gu, Alice Oh, Taekyung Kim
arXiv Machine Learning
Sep 7

Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills

The paper introduces the Neutral Prompting Attack (NPA), a stealthy method that uses semantically benign instructions to increase the likelihood of large language models hallucinating non‑existent package names in coding agents. Unlike traditional dependency steering, NPA does not target a specific package but shifts the model’s output toward more speculative names. Experiments across multiple LLMs show that NPA raises hallucination rates, affects pip install success, alters the distribution of hallucinated packages, and bypasses existing static‑analysis, LLM‑based, and agent‑based defenses.

By Chia-Yi Hsu, Chia-Mu Yu, Chun-Ying Huang, Jun Sakuma
arXiv Computation and Language
Sep 7

The Anatomy of an ASR Hallucination

The paper investigates why automatic speech recognition (ASR) systems sometimes generate fluent text that does not correspond to the input audio, a phenomenon termed hallucination. By examining two independently trained Conformer‑Large models—one using CTC and the other RNN‑T—under conditions of environmental noise and speaker‑background shift, the authors identify the final encoder stage as a critical boundary. Bypassing this final block leads to divergence on almost all utterances, while bypassing earlier blocks has minimal effect; at this stage, representations become compact, the decoder can read the text, and grapheme information becomes explicit, yet the output is garbled or repetitive rather than fluent fabrication. The study thus pinpoints a mechanistic precondition for hallucination—failure to produce adequately grounded output—though it does not fully explain naturally occurring hallucinations, and it highlights a consistent terminal‑stage dependency across decoder families and distribution shifts.

By Hamees Sayed, Apoorv Singh, Kumar Aman, Akshat Mandloi
arXiv AI
Sep 7

VICAL: Vicinal Consistency Alignment for Long-Tailed Visual Recognition

VICAL is a framework for long‑tailed visual recognition that focuses on reducing prediction variance rather than increasing expert diversity. It combines Self‑Consistency Learning, which smooths the loss landscape and mitigates overfitting on tail classes, with Deep Ensemble Distillation, which encourages low‑frequency semantic agreement across experts. Experiments on CIFAR‑LT, ImageNet‑LT, and iNaturalist 2018 demonstrate that VICAL consistently outperforms state‑of‑the‑art methods.

By Jiangang Zhu, Zheng Wang, Bin Zhu, Yi-Ping Phoebe Chen, Jingjing Chen
arXiv AI
Sep 7

Gradient-based Model Shortcut Detection for Time Series Classification

The paper "Gradient-based Model Shortcut Detection for Time Series Classification" investigates how deep neural networks (DNNs) can develop shortcut learning behaviors in time series classification tasks, where models may rely on spurious correlations in the training data. It introduces a gradient‑based detection method that identifies point‑based shortcuts without needing test data or clean training classes, and evaluates this approach on UCR time series datasets.

By Salomon Ibarra, Frida Cantu, Kaixiong Zhou, Li Zhang
arXiv Machine Learning
Sep 7

UniMate: One Unified Model to Animate Diverse Skeletons

UniMate is a unified foundation model that generates articulated motion for any skeleton from a rigged 3D asset and a text prompt, eliminating the need for test‑time optimization or per‑skeleton retraining. It uses a topology‑aware diffusion transformer that incorporates skeletal topology through graph‑aware attention bias, spectral rotary position embedding, and a global topological conditioner. Trained on the newly curated UniML3D dataset of 13,006 diverse motion sequences, UniMate outperforms existing baselines in quality, generalization, and efficiency, and supports zero‑shot cross‑topology transfer, in‑betweening, expansion, and text‑guided editing.

By Linzhan Mou, Jiahui Lei, Zhiyang Dou, Chenyue Cai, Chaoyue Song, Adam Finkelstein, Szymon Rusinkiewicz
arXiv Machine Learning
Sep 7

Advancing Subseasonal Forecasting with Machine Learning

The paper introduces Probabilistic Bias Correction (PBC), a machine learning framework that learns to correct historical probabilistic forecasts, thereby reducing systematic errors in subseasonal weather predictions. Applied to leading dynamical and AI models from ECMWF, PBC doubles the AI system’s modest subseasonal skill and improves the operationally-debiased dynamical model for most pressure, temperature, and precipitation targets. In ECMWF’s 2025 real‑time forecasting competition, PBC’s global forecasts ranked first across all weather variables and lead times, outperforming multiple operational and ensemble models.

By Hannah Guan, Soukayna Mouatadid, Paulo Orenstein, Judah Cohen, Haiyu Dong, Zekun Ni, Jeremy Berman, Genevieve Flaspohler, Alex Lu, Jakob Schloer, Joshua Talib, Jonathan A. Weyn, Lester Mackey
arXiv AI
Sep 7

A Removal Based Approach to Improve LLM Faithfulness at Test-Time

The paper proposes a test‑time method to enhance the faithfulness of large language model (LLM) explanations by removing concepts not credited in the model’s explanation before re‑querying the model. This approach targets incompleteness—omissions of influential factors—rather than unsoundness, and is model‑agnostic, requiring no changes to model weights. Experiments across two datasets and multiple model families show improved faithfulness compared to standard prompting and faithfulness‑encouraging prompts.

By Qinglan Luo, S M A Nahian, John Guttag, S. Mazdak Abulnaga, Katie Matton
arXiv AI
Sep 7

Diffusion Language Models for Mobile Edge Agentic AI: Foundations, Applications, and Challenges

Diffusion language models (DLMs) provide a non‑autoregressive approach for mobile edge agentic AI, refining tokens through iterative denoising instead of left‑to‑right decoding. They can update multiple uncertain tokens in parallel and use bidirectional context, allowing flexible quality‑latency trade‑offs and early exits that reduce response delay and communication overhead. The survey reviews DLM foundations, resource‑efficient architectures, training and inference acceleration, compression, deployment strategies, and discusses open issues such as long‑context management, split inference, and trustworthy execution.

By Chenqi Li, Minghui Min, Dusit Niyato, Wei Ni
arXiv AI
Sep 7

When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMs

The paper introduces MPS-Bench, a benchmark of 5,181 scenarios from 584 real-world images across 12 high-risk domains, each paired with a hidden user profile, to evaluate personalized safety in vision‑language models (VLMs). Eight leading VLMs were tested and found to almost always respond directly (86‑99%) without seeking missing context, scoring no higher than 2.6/5 on personalized safety. The authors identify a phenomenon called visual dominance, where visual information enters text representations early and suppresses textual risk signals, and propose PRISM, a lightweight input monitor that predicts when a query should be deferred, achieving 0.978 AUC and outperforming all tested models on the safety‑utility Pareto frontier.

By Edward Sun, Yuchen Wu, Zixian Ma, Eric Hanchen Jiang, Yijia Xiao, Xiaoyuan Yi, Ranjay Krishna, Wei Wang, Jindong Wang, Aylin Caliskan
arXiv AI
Sep 7

Large Language Models for HVAC Operations in Building Energy Systems: A Critical Review of Methods, Applications, and Deployment Readiness

The paper reviews 66 studies on large language models (LLMs) applied to HVAC operations in building energy systems, categorizing them by application and method families and evaluating their evidence realism and deployment readiness. It finds that most work focuses on building energy modelling, with only four studies reaching pilot-level evidence and none reporting sustained operational deployment. LLMs are currently best suited as semantic and workflow layers—such as point‑name normalisation and document‑grounded operator support—rather than autonomous HVAC controllers, and future research should target field‑validated benchmarks and safe, low‑latency LLM‑MPC/RL integrations.

By Alexander Neubauer, Tianzhen Hong, Han Li, Mengbo Yu, Amin Darbandi, Yannick F\"urst, Martin Kriegel
arXiv Machine Learning
Sep 7

Conformal Prediction for Offensive Security

The paper examines the underexplored use of Conformal Prediction (CP) in offensive security, noting that while CP has been applied defensively, its role in attacks is rarely documented. The authors present preliminary results in two offensive domains: Privacy‑Preserving Machine Learning and network traffic analysis. They aim to bridge the gap between CP’s defensive successes and its potential for facilitating attacks.

By Giovanni Cherubin
arXiv Machine Learning
Sep 7

Nested Inductive Bias Framework for SPD Manifold Learning

The paper introduces a Nested Inductive Bias framework that uses a two‑stage diffeomorphic composition to pull back non‑Euclidean target geometries onto symmetric positive definite (SPD) manifolds. This approach allows the construction of curvature‑aligned Riemannian classifiers that respect both matrix constraints and the intrinsic relational geometry of data. Empirical results on kinematic, signal processing, and synthetic benchmarks show that class separability degrades when metric curvature does not match the data distribution, and the authors also propose the Rational Conformal Metric (RCM) for robust vectorized architectures.

By Tushar Das
arXiv Machine Learning
Sep 7

A Fairness Audit of the Duckworth-Lewis-Stern Method: Format-Specific and Gender-Differential Bias, with an Interpretable Calibration Layer for Cricket Target Revision

The paper audits the Duckworth‑Lewis‑Stern (DLS) method, the standard for revising cricket scores after rain, using 8,150 international matches to generate 233,550 synthetic interruption scenarios. It finds two structured biases: a 137‑run prediction error range across match‑state buckets and a gender‑differential bias in ODIs, with women’s scores over‑predicted by an average of +7.63 runs versus +1.51 runs for men. The authors benchmark DLS against five modern machine‑learning models and introduce DLS‑Cal, a lightweight calibration layer that reduces overall bias by 31% in ODIs and 19% in T20Is, and a gender‑aware variant that nearly eliminates the residual bias for women’s ODIs.

By Soumyadeep Roy