AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,638 stories · RSS feed

arXiv Computer Vision
1d ago

UINO-FSS: Unifying Representation Learning and Few-shot Segmentation via Hierarchical Distillation and Mamba-HyperCorrelation

The paper introduces UINO-FSS, a unified framework for few-shot semantic segmentation that merges knowledge from DINOv2 and SAM into a single-encoder architecture. It employs coarse-to-fine multimodal distillation, a bottleneck adapter for embedding alignment, a meta-visual prompt generator, and a mask decoder, enhanced by Mamba-based 4D correlation mining. Experiments on PASCAL-5i and COCO-20i demonstrate state‑of‑the‑art performance in 1‑shot settings, achieving 80.6% mIoU on PASCAL-5i and 64.5% on COCO-20i.

By Wei Zhuo, Zhiyue Tang, Wufeng Xue, Hao Ding, Junkai Ji, Linlin Shen
arXiv AI
1d ago

Vision Paper: Why Distributed LLM Post-Training and Decentralized Alignment Are Now Essential

The paper argues that distributed learning—federated and decentralized—should be a core approach for large language model post‑training, rather than an optional privacy measure. It critiques current methods like RLHF and DPO for assuming centralized preference data, citing scientific, practical, and ethical flaws. The authors present a three‑pillar framework (privacy and data sovereignty, pluralism and representational equity, robustness and scalability), formalize technical challenges such as reference drift and preference heterogeneity, and outline algorithmic directions and a research agenda to make distributed post‑training feasible.

By Zhanhong Jiang, Qisai Liu, Soumik Sarkar
arXiv AI
1d ago

Does Context Compression Preserve Refusal Alignment?

The paper examines whether context compression, which encodes inputs into compact representations while preserving semantic content, maintains refusal alignment in instruction-tuned language models. It finds that encoder-based compression consistently weakens refusal behavior across various model families and architectures, even though reconstruction fidelity remains high. Mechanistic analysis attributes this to attenuation of activation along the decoder’s learned refusal direction, and introduces Memory Steering—a lightweight inference-time intervention that restores refusal rates without retraining.

By Anthony Nguyen, Wenjun Lin
arXiv AI
1d ago

Objective-Conditioned Active Preference Learning for Multi-Objective Alignment

The paper introduces OGC-DPO, an active preference learning framework for aligning large language models with multiple objectives such as helpfulness, harmlessness, and honesty. OGC-DPO treats each response pair and objective as a separate query, using the model’s implicit gradient to gauge informativeness and selecting queries that expand gradient coverage for each objective. Experiments show that OGC-DPO outperforms existing methods at the same annotation cost and reveals that informative conflicts can serve as valuable training signals.

By Duc Anh Vu, Thong Nguyen, Hoang Pham, Ponhvoan Srey, Luu Anh Tuan
arXiv AI
1d ago

VIPBench: A Human-Aligned Benchmark for Voice Identity Perception in the Age of Voice Cloning

VIPBench is a new benchmark that measures how humans perceive voice identity, using 124,876 same-or-different judgments from 1,290 listeners on 9,800 recording pairs involving real voices, voice clones, and morphed voices of 100 public figures. It defines four tasks to evaluate how well speaker‑representation models align with human judgments, including predicting majority responses and assessing similarity across real and synthetic recordings. Eleven publicly available speech models were evaluated, showing that listeners often misidentify cloned voices and that no model reaches the noise ceiling, with the best explaining only 61% of the explainable variance.

By Xudong Tang, Matthew Groh
arXiv AI
1d ago

Practical Estimation of the Bayes Optimal Fairness-Accuracy Tradeoff with Soft Labels

The paper presents a method to estimate the Bayes optimal fairness‑accuracy tradeoff using only soft labels and group information. It derives an explicit expression for the fair Bayes error rate and constructs consistent, instance‑free estimators with provable bias bounds and sample complexity guarantees. Empirical results show that the estimators provide a robust, low‑variance estimate of the optimal fairness‑accuracy Pareto frontier even when soft labels are noisy.

By Mohit Sharma, Okan Ko\c{c}, Amit Deshpande, Takashi Ishida, Gang Niu, Rajiv Ratn Shah, Masashi Sugiyama
arXiv AI
1d ago

Touch\'{e}World tech report : Towards Online Unified Camera--Hand Exploration in Generative Worlds

TouchéWorld is a framework that merges gesture-driven interaction with camera-controlled video world models, enabling users to explore generated environments using both camera navigation and continuous hand motion. Starting from an initial image and a scene-description prompt, the system generates video conditioned on temporally aligned camera and hand controls, built from joint camera–hand training data derived from heterogeneous egocentric videos. The approach incorporates spatial hand-skeleton conditioning, numerical hand-motion features, causal adaptation, and few-step distillation to support chunk-level online generation, while TouchéBench evaluates hand-motion following, camera–hand coordination, online interaction, and object response.

By Jiaao Yu, Yanjian Zhang, Haoyang Jiang, Guillaume Wisniewski, Nadi Tomeh, Shijie Jia
arXiv AI
1d ago

Reasoning-Aware Cross-Modal Association Unlearning in Multimodal Large Reasoning Models

The paper introduces Visual‑Keyed Counterfactual Rebinding (VKCR), a parameter‑efficient method for unlearning specific visual‑to‑knowledge associations in multimodal large reasoning models. VKCR isolates the association in a compact parameter subspace, suppresses target‑bearing outputs for the designated visual context, and counterfactually rebinds reasoning to favor valid, non‑sensitive alternatives. Evaluation with CLEAR‑R and MLLMU‑R shows that VKCR markedly reduces image‑conditioned target leakage and hallucinated reassociation while preserving normal visual reasoning and text‑only knowledge.

By Xinghao Li, Chenchen Tan, Youyang Qu, Shujie Cui, Xingliang Yuan, Cunjian Chen, Chi Liu, Longxiang Gao
arXiv AI
1d ago

CoSafe: Cooperative Runtime Safety Assurance for Mixed-Traffic Autonomous Driving

CoSafe is a modular, ADS-agnostic framework that extends runtime safety assurance from ego-centric takeover to scene-level cooperative intervention in mixed-traffic autonomous driving. It uses vehicle-to-everything communication to allow roadside units to monitor hazards and jointly mitigate risks involving multiple vehicles. Evaluation on the InterDrive benchmark shows that CoSafe improves driving scores by 70.03%, reduces safety violations by 87.80%, and accurately identifies unsafe autonomous vehicles with an F3 score of 0.930 while meeting real-time constraints.

By Dingji Wang, You Lu, Haowen Jiang, Yanggang Chen, Yuyang Bi, Bihuan Chen, Shenglin Bao, Xin Peng
arXiv AI
1d ago

From Admission to Retention: Predicting Durable Offspring in Quality-Diversity Search

The paper investigates how to predict which offspring in a Quality‑Diversity (QD) archive will be retained over time, rather than just entering the archive. By analyzing signals available before reproduction, the authors find that lineage depth is the most reliable indicator of long‑term persistence, though its predictive power varies with population size. Despite this insight, biasing parent selection toward these signals does not improve overall search performance and can even lower the QD‑score.

By Corrado Calamita, Matteo Mendicini, Luca Ferrari, Riccardo Bertamini
arXiv AI
1d ago

Trick-or-Treat: Adversarial Evaluation of Multimodal Medical Reasoning with Synthetic Images

Trick-or-Treat is an adversarial multimodal medical benchmark that pairs real chest radiographs with synthetic clinical photographs to test cross‑modal reasoning. Each condition has matched pairs where the radiograph and photograph suggest the same disease, and mismatched pairs where the photograph shows an alternative finding. In tests of three frontier models, hallucinations of clinically significant findings rose from 25‑37.5% for single photographs to 45‑57.5% when mismatched pairs were presented, with models often ignoring mismatches and generating findings that better aligned with the radiograph.

By Oishi Banerjee, Eujin Lee, Sung Eun Kim, Alexandra N. Willauer, Christine Y. Zhou, Zhizhou Yang, Pranav Rajpurkar
arXiv AI
1d ago

Population-Aligned Persona Generation for LLM-based Social Simulation

arXiv:2509.10127v3 Announce Type: replace-cross Abstract: Recent advances in large language models (LLMs) have enabled large-scale, high-fidelity social simulations, creating new opportunities for co...

By Zhengyu Hu, Jianxun Lian, Zheyuan Xiao, Max Xiong, Tianfu Wang, Teng Xiao, Yuxuan Lei, Fengqing Jiang, Kaize Ding, Ziang Xiao, Nicholas Jing Yuan, Xing Xie, Radha Poovendran