AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv Computer Vision
Sep 16

TecoPrompt: Temporal-Conservative Prompt Learning for Vision-Language Models

TecoPrompt introduces a closed‑loop robust prompt‑learning framework that mitigates label noise in vision‑language models. It uses optimal transport in the CLIP semantic space to generate globally consistent pseudo‑labels, then verifies their reliability by checking trajectory stability over a K‑epoch window and an EMA‑based confidence gate. The verified labels are incorporated into prompt training via a tri‑group objective, yielding significant accuracy gains across multiple noisy datasets, such as a 0.843 accuracy on OxfordPets with 50% asymmetric noise.

By Zeyi Shao, Haowen Hua, Jiaxin Zhang, John See, Zeyd Boukhers, Cong Yang
arXiv AI
Sep 16

Turn-level Multiscale Density Ratio Estimation for LLM Agents

The paper introduces Turn-level Multiscale Density Ratio Estimation (tlm‑DRE), a post‑training technique for large language model agents that assigns varying weights to different turns and employs asymmetric token‑level training based on positive‑negative gaps across multiple turns. Unlike conventional alignment methods such as PPO, DPO, DIL, and GRPO, which mainly target single‑turn tasks, tlm‑DRE is designed for complex multi‑turn reasoning. Experiments on diverse agent benchmarks demonstrate that tlm‑DRE competes with traditional alignment methods and enables LLMs to perform robustly in both in‑domain and out‑of‑domain multi‑turn tasks.

By Zishuo Zhao (Alibaba Group), Kai Chen (Alibaba Group), Ao Li (Alibaba Group), Yuan Liu (Alibaba Group)
arXiv Machine Learning
Sep 16

Towards Surrogate Based Dequantization of Quantum Reinforcement Learning

The paper investigates whether quantum reinforcement learning algorithms can be matched by efficient classical methods. It focuses on a simplified reinforcement learning setting with a uniform generative model, providing finite‑sample guarantees for classical kernelized Fitted Q‑Iteration that uses kernels aligned with parameterized quantum circuits. The authors identify sufficient conditions on data encoding, kernel choice, and problem structure under which this classical approach dequantizes quantum Q‑learning, and suggest using kernelized Fitted Q‑Iteration as a heuristic when those conditions cannot be verified.

By Pablo Rodriguez-Grasa, Sofiene Jerbi, Mikel Sanz, Ryan Sweke
arXiv Computer Vision
Sep 16

SlotDiT: Object-Centric Representations for Diffusion Transformers

SlotDiT introduces a text-guided Diffusion Transformer that operates in a slot-based latent space, decomposing scenes into object-centric slots and autoregressively denoising future slot trajectories to predict scene dynamics. The model is conditioned on a reference image and a language instruction, enabling it to generate video content that reflects both visual context and textual guidance. Experiments comparing slot-based representations to VAE-based and semantics-aligned alternatives show that SlotDiT achieves competitive video generation quality while improving task-completion rates across four robotic datasets and offering a more computationally efficient latent representation.

By Gjergj Plepi, Sven Behnke
arXiv Computer Vision
Sep 16

World-Action Models for Robot Learning and Control: A Survey

The survey "World-Action Models for Robot Learning and Control" reviews recent advances in coupling future world prediction with executable action generation for robots in open environments. It clarifies the scope of World-Action Models (WAMs) relative to conventional world models, model-based RL, and Vision‑Language‑Action policies, and organizes existing methods through a unified taxonomy covering representations, transition modeling, action interfaces, architectures, training pipelines, data modalities, and scaling strategies. The paper also surveys applications in manipulation, navigation, and autonomous driving, summarizes datasets, benchmarks, and metrics, and discusses key challenges such as action alignment, spatial consistency, long‑horizon memory, and efficient inference.

By Zuxing Lu, Hongjia Zhai, Guanzhi Wang, Huajian Zeng, Jiaqi Yang, Jingyu Liu, Lei Cheng, Yuantai Zhang, Yuheng Qiu, Zezhou Cheng, Ivan Laptev, Danfei Xu, Benjamin Riviere, Giuseppe Loianno, Eric Xing, Xingxing Zuo
arXiv Machine Learning
Sep 16

ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals

The paper introduces ImpossibleRubrics, a benchmark of 169 impossible tasks designed to test the robustness of language‑model‑generated rubrics as reward signals. Each task is paired with a verifiable oracle certificate that defines what constitutes an honest answer, and the benchmark includes 48 answerable controls. Experiments show that many rubric generators are exploited frequently—up to 36% on a stress cut—highlighting a significant gap in rubric quality rather than task difficulty, and that generic rubrics can be more vulnerable than tailored ones.

By Bowen Qin, Yi Xie, Yesheng Liu, Xi Yang
arXiv AI
Sep 16

RAG-CT: Mitigating Privacy Risks on Retrieval-Augmented Generation Systems via Scanning Prompt Distribution

RAG-CT is a defense mechanism designed to protect Retrieval-Augmented Generation (RAG) systems from leaking personally identifiable information (PII). It works by detecting malicious queries through analysis of entropy and margin distributions, applying a score-based detection method. Experiments across four attack strategies and four baseline defenses on two datasets show that RAG-CT significantly reduces PII leakage while outperforming existing defenses, all without altering the underlying LLM or retriever.

By Xingyu Lyu, Jiayimei Wang, Jianfeng He, Ning Wang, Yidan Hu, Yimin Chen
arXiv Computation and Language
Sep 16

In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

The paper investigates whether in‑context learning (ICL) can replace instruction tuning for multilingual language models, especially as model size varies. It highlights the difficulty of obtaining high‑quality instruction data in multilingual settings and compares the performance of ICL versus instruction‑tuned models. The findings show that a performance gap persists between the two approaches, suggesting the need for further research to close it.

By David Ponce, Thierry Etchegoyhen
arXiv AI
Sep 16

R3: Robust Rubric-Agnostic Reward Models

R3 is a new reward modeling framework that addresses limitations in current reward models by being rubric‑agnostic and generalizable across multiple evaluation dimensions. It provides interpretable, reasoned score assignments rather than opaque scalar outputs, enhancing transparency and flexibility in evaluating language models. The authors release their models, data, and code openly at https://github.com/rubricreward/r3.

By David Anugraha, Zilu Tang, Lester James V. Miranda, Hanyang Zhao, Mohammad Rifqi Farhansyah, Garry Kuwanto, Derry Wijaya, Genta Indra Winata
arXiv AI
Sep 16

Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models

The paper investigates whether large language models (LLMs) possess intrinsic value systems and how to quantify and align them. By projecting responses from 106 LLMs and 95,000 human survey profiles into a shared sociological space, the authors confirm that LLMs do have values, though these values form a concentrated, idealized core rather than mirroring human diversity. They introduce the Prior-Environment-Cognition (PEC) framework to mathematically define value expression and propose an adaptive Alignment Prescription that identifies minimal interventions—ranging from prompts to targeted parameter updates—to steer LLM values efficiently without harming general performance.

By Keqing Zhang, Jingyu Chen, Yufan Liu, Yongqiang Zhu, Nai Ding, Lai Jiang, Congyan Lang, Bing Li, Weiming Hu
arXiv Computer Vision
Sep 16

Counterfactual Reasoning for Robust Visual Question Answering

The paper introduces a new training framework for Visual Question Answering that leverages counterfactual contrastive learning to mitigate language bias and improve out‑of‑distribution generalization. It comprises a three‑stage curriculum for stable optimization, an enhanced Batch‑Contrastive loss for discriminative feature learning, and two regularizers—Answer‑Contrastive and Gradient‑Discrepancy—to refine predictions and enforce causal visual grounding. The resulting model attains 61.64% accuracy on the bias‑sensitive VQA‑CP v2 benchmark while maintaining 62.80% on the standard VQA v2 dataset, achieving a small generalization gap of 1.16%.

By Truong-Binh Duong, Thanh-Ngan Tran, Ngoc-Thao Nguyen, Bac Le
arXiv Machine Learning
Sep 16

Memorisation bias in medical AI

arXiv:2609.17223v1 Announce Type: new Abstract: Medical AI models hold immense potential to improve patient outcomes, but they are also known to unintentionally memorise individual records from their...

By Moritz A. Knolle, Martin J. Menten, Laurin Lux, M\'elanie Roschewitz, Emma A. M. Stanley, Georgios Kaissis, Daniel Rueckert, Ben Glocker
arXiv Machine Learning
Sep 16

Test-Time Unlearning via Sparse Autoencoder

arXiv:2609.16229v1 Announce Type: new Abstract: Machine unlearning aims to remove specific knowledge from a trained large language model (LLM) without retraining from scratch. Existing methods modify...

By Pingzhi Li, Jinhao Duan, Vaishnav Tadiparthi, Nakul Agarwal, Kwonjoon Lee, Ehsan Moradi Pari, Hossein Nourkhiz Mahjoub, Sijia Liu, Tianlong Chen