AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv Machine Learning
Sep 17

Beyond Direct Sensing: Harnessing Indirect Observations from Third-Party Sensors in Vehicle Tracking

The paper introduces GrayTrack, a vehicle‑tracking system that fuses weak, indirect observations from third‑party sensors with sparse direct sensor data using a road‑constrained particle filter. Experiments on a CARLA‑Mininet‑WiFi pipeline show that the system achieves an F1 score of 0.989 for anonymous vehicle passages and reduces trajectory RMSE by 60.1% while cutting catastrophic track loss from 35.8% to 0.3%. These results demonstrate that incorporating indirect third‑party observations can substantially extend tracking capabilities when direct sensor access is limited.

By Gaofeng Dong, Vamsi Eyunni, Pragya Sharma, Kang Yang, Mani Srivastava
arXiv Machine Learning
Sep 17

Patch the Distribution Mismatch: RL Rewriting Agent for Stable Off-Policy SFT

The paper introduces a reinforcement‑learning based rewriting agent that rewrites training data to reduce the distribution mismatch between supervised fine‑tuning (SFT) and a language model’s generation distribution. By formulating data rewriting as a policy‑learning problem, the authors train a lightweight LoRA rewriting policy that optimizes alignment with question‑answering style, maintains semantic diversity, and enforces task consistency. Experiments on three instruction‑tuned backbones show that models fine‑tuned with the rewritten data achieve downstream performance comparable to standard SFT while mitigating degradation on non‑downstream benchmarks, and preliminary tests suggest the policy can transfer across domains such as logical reasoning and medical question answering.

By Jiacheng Wang, Zhijie Liu, Ping Jian, Zirong Chen, Ke Ren Liao, Zhen Yang, Zhongbin Guo
arXiv Machine Learning
Sep 17

Correcting Boundary Bias and Observation Independence in Bayesian Experimental Design

The paper tackles two shortcomings of Gaussian‑process based active learning: (1) the posterior variance is independent of observed values, reducing sensitivity to data structure, and (2) it over‑inflates variance near domain boundaries, causing excessive edge sampling. The authors propose a reconstruction‑driven design density that warps sampling toward regions where the posterior mean changes rapidly, and a geometric equalizer that corrects boundary bias. Experiments on sixteen synthetic and two real‑data benchmarks show that the equalizer consistently improves function reconstruction, while the warp further enhances performance by concentrating measurements where the target function varies most.

By Sanna Jarl, Jens Sj\"olund, Jonathan J. S. Scragg, Maria B{\aa}nkestad
arXiv AI
Sep 17

How to Compress KV Cache in RL Post-Training? Shadow Mask Distillation for Memory-Efficient Alignment

The paper addresses the memory bottleneck in reinforcement learning for large language models caused by the large Key-Value (KV) cache during rollout phases. It highlights that while KV cache compression can reduce memory usage, it introduces a significant off‑policy bias that standard statistical corrections cannot adequately mitigate. The authors argue that even tiny compression errors are amplified by RL’s instability, leading to inefficient learning.

By Rui Zhu, Weiheng Bai, Qiushi Wu, Yang Ren, Haixu Tang, Yuchu Liu
arXiv Machine Learning
Sep 17

Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs

The paper introduces the Tri‑Metric Router, a deterministic, training‑free policy that chooses among Raw, Neural, and Lexical pipelines for retrieval‑augmented generation on commodity GPUs. It uses three CPU‑side signals—spatial complexity, syntactic density, and type‑token ratio—to balance VRAM headroom and latency, calibrated on LongBench qasper. The method eliminates out‑of‑memory failures and improves alignment and F1 scores compared to always‑on lexical compression without extra VRAM or training costs.

By Saipraveen Vabbilisetty, Ajay Kumar Boddepalli, Deep Narayan Mishra, Shashank Kapadia, Haoan Wang, Anupriya Sharma
arXiv AI
Sep 17

Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces

The paper introduces label‑free bias‑only test‑time reinforcement learning (TTRL), which uses majority‑vote pseudo‑labels as rewards and optimizes only about 100 K bias parameters while keeping the pretrained backbone frozen. On the MATH‑500 benchmark it achieves 76.67 % accuracy, slightly better than a labeled bias‑steering baseline, and improves performance on several vision‑language and audio reasoning tasks. The authors also show that the learned steering vectors transfer to 4,500 held‑out MATH problems and analyze why such a highly restricted adaptation works, linking majority‑vote reliability to rollout consensus and gradient energy in bias subspaces.

By Naveen Vakada, Mingyuan Li, Shaoxiong Ji
arXiv AI
Sep 17

Using OCR Heads to Verbalize Image Semantics

The paper investigates how vision‑language models (VLMs) perform optical character recognition (OCR) by identifying attention heads that are causally necessary for OCR across four models. These heads are shown to be general‑purpose, producing interpretable semantic features for any image token, such as recognizing the word "bike" or the concept "feathers". By collapsing the heads’ attention weights into a verbalization lens transformation, the authors reveal that image representations align with language from early layers and can even be used to edit non‑word concepts in images, demonstrating the broader utility of this subspace.

By Sheridan Feucht, Benno Krojer, Sarah Wang, Henry Abrahamsen, Byron C. Wallace, David Bau
arXiv Machine Learning
Sep 17

FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learning

FedGuide is a federated reinforcement learning framework that tackles distributional mismatch among heterogeneous clients by using diffusion priors as personalized behavior models. It aggregates these priors via an Optimal-Transport Mixture-of-Experts, preserving diverse behavior modes, and introduces a Distribution Correction Estimation (DICE) value baseline to give low‑variance, return‑aware guidance for local policy improvement. Experiments on heterogeneous environments demonstrate that FedGuide surpasses existing FRL methods in client‑average returns, final‑round performance, and worst‑round robustness while maintaining stable learning under stronger heterogeneity.

By Zhilin He, Gauri Joshi
arXiv AI
Sep 17

Making AI-Assisted Claims Independently Challengeable: Publication Authority and a Protocol for Falsifiable Publication Records

The paper introduces Publication Authority, a single-use, non-transferable capability that ensures AI-assisted claims can be independently challenged by providing a machine-readable, falsifiable publication record. It presents the PAC-2026 protocol, evaluates its fourth bounded semantic freeze (SF-4), and demonstrates through extensive modeling that the system enforces strict obligations on evidence, authorization, and lifecycle continuity. The study confirms internal coherence, bounded safety, and fault sensitivity, though it does not address factual truth or field efficacy.

By Torsten Olivi Tiltack, Yifei Dong, Kun Yu, Xu Wang, Wei Liu, Jianlong Zhou, Ren Ping Liu, Fang Chen
Simon Willison
Sep 16

Quoting Mustafa Suleyman

Mustafa Suleyman argues that artificial models should not be treated as if they possess feelings, preferences, rights, or any entitlement to human welfare. He emphasizes that consciousness underpins our ethical, legal, and political frameworks, and extending such rights to AI would lack evidence and complicate containment and alignment efforts.