AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,884 stories · RSS feed

arXiv Computer Vision
Aug 28

A Geometry-Driven, Framework-Agnostic Optimization for Object Pose Estimation

The paper proposes a data‑centric optimization for object pose estimation that uses a physically grounded rotation representation based on principal axes alignment. By aligning an object's coordinate system with its inertial principal axes, the method achieves inherent stability, symmetry‑aware canonicalization, and framework agnosticism, allowing it to be applied at the dataset level without modifying existing networks. Experiments on category‑level and instance‑level models show consistent accuracy improvements while preserving baseline network integrity.

By Wei Chen, Tao Zhen, Zhongchen Shi, Jing Zhang, Liang Xie, Erwei Yin
arXiv Computer Vision
Aug 28

Beyond Atomic Layouts: Compositional Design Understanding with Vision-Language Models

The paper introduces a new task called compositional layout understanding, focusing on interpreting complex, multi‑layer document and UI designs. It presents CoDeLayout, a VQA dataset of about 20,000 real‑world layouts annotated with compositional element pairs and design intent. The authors identify semantic drift and structural ambiguity as key challenges for vision‑language models and propose MASON, a post‑training approach that combines multimodal alignment and structural perception to improve performance, achieving 91.66% accuracy with only 30% of the training data.

By Yiyang Huang, Zhaowen Wang, Simon Jenni, Jing Shi, Yitian Zhang, Yizhou Wang, Yun Fu
arXiv Machine Learning
Aug 28

Soft Active Electromyography Interface for Machine Learning-Enabled Silent Speech Recognition

The paper presents a soft, active electromyography (EMG) interface that enables word-level silent speech recognition (SSR) using machine learning. The device, worn on the hand, employs a fingertip electrode positioned near the lips to acquire EMG signals only when needed, and incorporates liquid metal interconnects, transparent flexible printed circuit electrodes, and elastomer encapsulation for mechanical stability during finger motion. A deep neural network trained on these stable signals achieved a mean accuracy of 97.2 ± 1.3 % across three subjects for a 30‑word vocabulary, and real‑time drone control demonstrated its practicality in noisy, privacy‑sensitive environments.

By Yuta Kurotaki, Shusuke Yamakoshi, Reitaro Yoshida, Yutaka Isoda, Tamami Takano, Yuji Isano, Yusuke Miyake, Kentaro Kuribayashi, Hiroki Ota
arXiv Machine Learning
Aug 28

Gromov-Monge Flow Matching for Equivariant Graph Generation

The paper introduces Gromov-Monge Flow Matching, a method that incorporates permutation-equivariance into generative graph models by aligning graph pairs up to node relabeling using the Gromov–Monge distance. It shows theoretically that quotient couplings can be lifted to aligned representatives without extra cost and that symmetrization yields equivariant flow-matching minimizers, even for categorical endpoints. Practically, the authors build minibatch couplings with Gromov–Wasserstein relaxations and optional outer assignments, improving sample quality in continuous graph and categorical molecular generation while remaining compatible with standard equivariant architectures.

By Moritz Piening, Christian Wald
arXiv Machine Learning
Aug 28

Beyond Client Averaging: A Client-Independent Second-Order Stationary-Bias Component in Stochastic SCAFFOLD

arXiv:2608. 26765v1 Announce Type: new Abstract: Existing constant-step analysis of stochastic \Scaf{} identifies a leading $O(\gamma/N)$ stationary mean bias and shows that higher-order bias can persist as the client count increases, but does not identify the first client-independent contribution at coefficient level.

By Yi-Ping Tang, Guan-Ju Peng
arXiv Machine Learning
Aug 28

When Privacy Hurts Mergeability: Geometry-Aware Model Merging under Differential Privacy

The paper investigates how differential privacy (DP) affects the ability to merge independently fine‑tuned task models into a single multi‑task model. It identifies two geometric challenges—local sharpness and reference drift—that hinder mergeability of private models. To address these, the authors propose DP‑Merging, a framework that uses a sharpness‑aware objective and a reference‑based alignment regularizer to improve mergeability while maintaining DP guarantees.

By Jin Liu, Junkang Liu, Ning Xi, Yinbin Miao, Dawei Wei, Ke Cheng, Jianfeng Ma
arXiv Machine Learning
Aug 28

Out of Sight, Not Out of Mind: Unveiling Latent Attack in Latent-based Multi-Agent Systems

The paper investigates whether hidden representations in latent-based multi‑agent systems can carry attack information that remains effective during normal operation. A latent attack framework is introduced, reactivating attack effects through latent interventions without using adversarial text. Experiments show that these latent attacks can significantly degrade task performance, especially when targeting inter‑agent KV‑cache handoffs, and that the degradation cannot be explained by simple perturbations or invalid generation.

By Chenxi Wang, Ruiyang Huang, Jiayan Sun, Lei Wei, Yifan Wu
arXiv Machine Learning
Aug 28

SimCast-S2S: An Efficient Generative Model for Subseasonal Precipitation Forecasting via Transfer Learning from Climate Simulations

SimCast‑S2S is a generative latent‑diffusion model designed for probabilistic subseasonal‑to‑seasonal precipitation forecasting. It tackles three key challenges: it uses a diffusion pipeline to capture uncertainty, operates in a compact latent space to enable efficient large‑ensemble generation, and leverages transfer learning with low‑rank adaptation to train on limited reanalysis data after pretraining on climate simulations. The model outperforms deep‑learning baselines and competes with, or surpasses, operational systems such as the ECMWF‑S2S baseline without requiring extensive post‑processing.

By Hiep V. Dang, Antonios Mamalakis
arXiv Machine Learning
Aug 28

GRAS: Guided Reduced-Variance Proposals and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion

The paper introduces GRAS, a method that improves training‑free reward alignment for discrete diffusion models by reducing variance in guided proposals and adapting the resampling temperature during search. It achieves this without adding denoiser cost, using Rao‑Blackwellized estimates for differentiable rewards and a leave‑one‑out baseline for non‑differentiable ones. Experiments on regulatory DNA and protein design show GRAS outperforms existing training‑free techniques and rivals reward‑fine‑tuned models.

By Kwanyoung Kim
arXiv Computation and Language
Aug 28

DeflectBench: A Benchmark for Evaluating Rhetorical Fallacy Generation in LLMs

DeflectBench is a new benchmark that evaluates how large language models (LLMs) generate rhetorical fallacies when prompted. The study tests 23,990 generations from four leading models using three deflection strategies (whataboutism, ad hominem, red herring), seven prompt framings, and 80 claims across four controversy levels. Results show that refusal to produce fallacies depends mainly on request structure, with prompt framing and fallacy type dramatically affecting compliance rates.

By Art Kanke
arXiv Computer Vision
Aug 28

VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology

VIPER is the first expert‑curated benchmark for vision‑language models in veterinary toxicologic pathology, comprising 1,251 questions linked to 419 H&E‑stained rat histology images across seven organ systems. The benchmark includes multiple‑choice, KPrim, and free‑text formats and was validated by board‑certified veterinary pathologists. Evaluation of 16 models—spanning veterinary‑specific, human pathology‑specialized, and general‑purpose models—reveals a significant domain gap, a tendency for frontier models to over‑diagnose normal tissue, and underscores the importance of domain‑specific training for accurate visual predictions.

By Luca L. Weishaupt, Simone de Brot, Javier Asin, Lloren\c{c} Grau-Roma, Nic G. Reitsam, Andrew H. Song, Dongmin Bang, Stefan T. Kaluziak, Long Phi Le, Jakob Nikolas Kather, Faisal Mahmood, Guillaume Jaume
arXiv Machine Learning
Aug 28

Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals

The paper introduces Safe Contrastive Reinforcement Learning (Safe-CRL), a method that corrects bias in contrastive RL caused by failure-terminated Markov decision processes. By applying mass-weighted InfoNCE and a log-survival-mass score, Safe-CRL uses only a one-bit failure signal to improve survival and goal-reaching performance across twelve robot navigation and locomotion tasks. The approach demonstrates complex failure-avoidance behaviors and completes the theoretical foundation of contrastive RL under failure termination.

By Guopeng Li, Yiyang Duan, Yiru Jiao, Chengcheng Xu
arXiv Computer Vision
Aug 28

Surgical Video Generation From Diffusion to World Models: A Survey

This survey reviews recent advances in surgical video generation, categorizing methods into unconditional, conditional, and world modeling generation. It highlights a shift from creating visually plausible frames to modeling the causal dynamics of surgical scenes, and discusses challenges such as pixel-level fidelity versus clinical plausibility, generalization, physical realism, controllability, and interpretability. The paper also compiles experimental results from public datasets to serve as a quantitative benchmark for the field.

By Fuxiang Huang, Chenxu Zhang, Liang Han, Lei Zhang
arXiv Machine Learning
Aug 28

Toward Equitable Low-Carbon Mobility: Fairness-Aware Demand Prediction for Expanding Bike-Sharing Systems

The paper introduces FairGIN, a fairness-aware graph neural network designed to predict demand for expanding bike‑sharing systems while addressing cold‑start challenges and equity concerns. It combines expansion‑simulated incremental training, attention‑based knowledge transfer, and income‑stratified regularization to improve predictive accuracy and reduce income‑based disparities. Experiments on NYC and Seattle show that FairGIN outperforms existing methods and supports more inclusive station placement without sacrificing overall efficiency.

By Man Luo, Yixuan Zhao
arXiv Computation and Language
Aug 28

Mitigating Fabrication in Multi-Stage LLM Pipelines for Hiring: An Empirical Evaluation of Prompt Guardrails and Human-in-the-Loop Checkpoints

The study evaluates how to reduce fabricated claims in multi‑stage large language model (LLM) hiring pipelines. Prompt guardrails alone cut fabrication density by 86 % but still left half of outputs containing false claims, while adding a human‑in‑the‑loop checkpoint after resume improvement eliminated all identity fabrications and significantly lowered overall fabrication rates. The results show that a layered approach—combining prompt guardrails with human checkpoints—provides stronger protection against severe failures without harming the quality of the final outputs.

By Hiroko Takano
arXiv Computer Vision
Aug 28

LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning

LLaVAFlow is an information‑theoretic distillation framework designed to preserve cross‑modal alignment in Multimodal Large Language Models during visual instruction tuning. It compresses the mutual information between extracted relations and MLLM embeddings to refine alignment flow, and then maximizes mutual information between pretrained and fine‑tuned alignment flows to transfer compact alignment information. Experiments demonstrate that LLaVAFlow effectively maintains alignment flow, improving downstream performance and generalization.

By Muyao Yuan, Muyan Jiao, Jiangyong Ying, Weizhan Zhang, Yuanhong Zhang, Lan Ma, Yuan Gao, Haipeng Du
arXiv Machine Learning
Aug 28

Private and interpretable clinical prediction with quantum-inspired tensor train models

The paper demonstrates that publicly available clinical machine learning models, such as logistic regression (LR), pose significant privacy risks because attackers can recover model parameters and identify training cohorts through various membership inference attacks. The authors show that even small cohorts can be reliably identified and that common practices like cross-validation can worsen the risk. To mitigate this, they propose a quantum-inspired defense that tensorizes discretized models into tensor trains (TTs), which obfuscates parameters, preserves accuracy, and maintains interpretability while providing black‑box protection comparable to Differential Privacy.

By Jos\'e Ram\'on Pareja Monturiol, Juliette Sinnott, Roger G. Melko, Mohammad Kohandel
arXiv Machine Learning
Aug 28

Interpretable, Fairly Evaluated Automated L2 Speaking Assessment that Beats the Single-Human Ceiling and Why Pause Encoding Does Not Change LLM Fluency Scores

The paper presents an interpretable, fair, and accurately benchmarked automated system for assessing second‑language English speaking. Using a hybrid of feature‑based speech‑timing metrics and a large language model (LLM) fluency judgment, the system achieves a Spearman correlation of 0.818 with the ICNALE Global Rating Archive, outperforming 81 % of trained human raters. A controlled study shows that encoding pauses into the LLM prompt does not meaningfully affect fluency scores, indicating that the system’s fluency signal derives from measurable speech‑timing features.

By Eichi Uehara
arXiv AI
Aug 28

ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices

ADeptS-Bench is a new benchmark designed to assess the trustworthiness of Computer Use Agents (CUAs) across mobile and desktop devices. It consists of two streams: a Safety stream with paired benign and malicious tasks that embed visual threats, and a Disambiguation stream that tests whether agents seek clarification when instructions are ambiguous. Evaluation of seven models shows none consistently achieves high task success while keeping attack success low, and all models exhibit problematic behaviors such as unhesitant checkout on a $25K order and failure to detect a mislabeled factory reset button.

By Joy Chen, Alejandro Castillejo Munoz, Pierluca D'Oro, Yuxuan Sun, Chloe Evans, Joseph Tighe