AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,884 stories · RSS feed

arXiv Computer Vision
Sep 1

Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

arXiv:2604.22280v4 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) have emerged as a promising foundation for universal multimodal embeddings. Recent studies have shown that...

By Peixi Wu, Ke Mei, Feipeng Ma, Bosong Chai, Zhibin Lan, Chenxi Zhao, Shannan Yan, Jie Chen, Zhangchi Hu, Yansong Peng, Bo Lin, Junjie Zhou, Dacheng Yin, Tianyi Wang, Fengyun Rao, Jing Lyu, Hebei Li, Xiaoyan Sun
arXiv Computer Vision
Sep 1

PIU: Proximity-guided Identity Unlearning in ID-Conditioned Diffusion Models

arXiv:2605.22311v2 Announce Type: replace Abstract: Identity-conditioned diffusion models enable high-quality and identity-consistent face generation, but they also raise severe privacy concerns, as...

By Jose Edgar Hernandez Cancino Estrada, Mauro D\'iaz Lupone, \v{Z}iga Emer\v{s}i\v{c}, Vitomir \v{S}truc, Peter Peer, Darian Toma\v{s}evi\'c
arXiv AI
Sep 1

SafeAtlas-VL: Beyond Binary Multimodal Safety with Large-Scale Data and Guard Models

SafeAtlas-VL introduces a large multimodal safety dataset with 1.5 million instances, rating image, request, and response risks on a five‑level ordinal scale across 15 harm categories and 55 subcategories. The accompanying SafeAtlas‑Bench provides 5,000 held‑out cases for evaluating ordinal predictions and continuous risk scores. Models trained on this data, including an 8B Guard model, achieve state‑of‑the‑art performance, outperforming prior benchmarks by about 4% in F1 score.

By Zongrui Wang, Xiangyang Zhu, Sicheng Wang, Han Wang, Dingyi Rong, Zeyu Zhang, Chunyi Li, Yue Shi, Kaiwei Zhang, Zicheng Zhang, Yuan Tian, Qi Jia, Yan Teng, Wei Sun, Ning Liu, Guangtao Zhai
arXiv AI
Sep 1

CoLa-ICD: A Knowledge-Enhanced Framework for Long-Tail Automated Medical Coding

CoLa-ICD is a knowledge‑enhanced framework designed to improve automatic medical coding of ICD codes in long, imbalanced clinical documents. It enriches ICD labels with external terms, models dependencies among related codes, and strengthens the alignment between label semantics and clinical evidence, particularly for rare codes. Experiments demonstrate that CoLa-ICD achieves state‑of‑the‑art performance in AUC, F1, and P@k, with larger gains in larger and sparser label spaces.

By Yihang Cheng, Veronica Liesaputra, Andrew Trotman
arXiv AI
Sep 1

TPR-Attention for Combinatorial Generalization

The paper introduces TPR-Attention, an attention mechanism that operates over tensor‑product representations to embed structured inductive bias into deep learning models. Experiments on compositional tasks demonstrate that TPR‑Attention outperforms existing architectural components in achieving combinatorial generalization. The results suggest that incorporating explicit compositional structure into neural attention can improve systematic generalization.

By Melisa Civeleko\u{g}lu, Isabeau Pr\'emont-Schwarz
arXiv AI
Sep 1

Preference Shapes Relevance: Cross-component Hierarchical Semantic Alignment for Personalized Generative Retrieval

The paper introduces CHAP, a personalized generative retrieval framework that aligns query semantics with item representations through a hierarchical semantic alignment module and models user behavior using both discrete Semantic IDs and continuous representations. It also proposes a Residual Cascading Generation mechanism to reduce inference latency by limiting the Transformer decoder to a single pass. Experiments on multiple datasets and online A/B tests show that CHAP outperforms existing methods, demonstrating its practical value.

By Gaoming Zhang, Angqing Jiang, Jianchun Song, Kena Qi, Dayao Chen, Wei Lin, Defu Lian
arXiv AI
Sep 1

Reverse N-Wise Output-Oriented Testing for AI/ML and Quantum Computing Systems

The paper proposes reverse n‑wise output‑oriented testing for AI/ML and quantum computing systems, a method that builds covering arrays over output equivalence classes, confidence buckets, decision boundaries, fairness partitions, embedding clusters, ranking stability bands, quantum measurement distributions, and error syndrome patterns. It then uses gradient‑free metaheuristic optimization to solve the inverse mapping problem, generating input configurations or quantum circuit parameters that trigger specific behavioral signatures in opaque models. The framework claims to provide explicit coverage guarantees, higher fault detection rates for calibration, boundary, and error syndromes, improved test suite efficiency, and automated partition discovery for MLOps and quantum validation pipelines.

By Lamine Rihani
arXiv AI
Sep 1

Adversarial Trust Poisoning in Vehicular Collaborative Perception

The paper introduces TrustFlip, an attack that exploits consistency‑based defenses in vehicular collaborative perception by deploying physical adversarial objects to create inconsistent observations among benign vehicles. This misattribution lowers the trust score of a targeted vehicle, leading to its exclusion from the collaboration and a degradation of perception performance. The authors evaluate the attack across multiple architectures, showing it can remove a benign vehicle in up to 87.7% of scenarios and reduce Average Precision by up to 13%, and propose a mitigation called TrustReflect that reduces the attack success rate by 35–100%.

By Yutong Liu, Chenyi Wang, Ming F. Li, Qingzhao Zhang
arXiv Computation and Language
Sep 1

Whose Assessment of Distress? Community Perspectives and LLM Alignment on Well-Being Posts

The study investigates how large language models (LLMs) assess psychological distress in online posts from six identity‑based communities. Through a perspectivist annotation task, 321 participants provided 9,587 judgments on 1,198 Reddit posts, revealing modest in‑group agreement (OR = 1.18) that varies across communities. When evaluated against these community‑specific labels, open‑weight LLMs consistently over‑estimate distress—achieving only 31–44% accuracy on posts perceived as none‑to‑mild—while newer models like GPT‑5 and Gemini 2.5 Pro show similar inflation, whereas Claude Opus 4 is more conservative. "whyItMatters":"The findings highlight that miscalibrated distress detection by LLMs can disproportionately impact the very communities they aim to serve, underscoring the need for equitable AI deployment in mental‑health contexts."

By Andrew Aquilina, Xiang Lorraine Li, Yu-Ru Li
arXiv Computation and Language
Sep 1

ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

The paper introduces ACTD, an Anchor-Based Cross-Tokenizer Distillation method that aligns vocabularies and sequences to transfer reasoning capabilities from large language models to smaller students. It uses a novel anchor loss with residual regularization to reduce alignment noise and extends the approach to multiple teachers. Experiments on five reasoning benchmarks with three teachers show state‑of‑the‑art results, with the multi‑teacher variant outperforming existing baselines.

By Huiyi Zhang, Zijian Li, Xiaocheng Feng, Weitao Ma, Xiaoliang Yang, Yichong Huang, Bing Qin
arXiv AI
Sep 1

SHADOWBENCH: Toward Reliable Automatic Evaluation of Semantic Alignment in Autoformalization

arXiv:2608.29270v1 Announce Type: cross Abstract: Autoformalization translates informal mathematical theorems into code for proof assistants such as Lean. A central challenge is that current evaluati...

By Hojae Han, Jongyoon Kim, Sanghyuk Park, Dongwook Cheon, Myungjae Jeon, Sunjong Choi, Soonho Kong, Wonseok Heo, Seung-won Hwang, Donghoon Hyeon