AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Computation and Language
Oct 23

Towards User Simulators That Stay On Task And Still Sound Natural

The paper introduces UserLM‑2.0, a reinforcement‑learning based user simulator that balances task intent adherence with conversational naturalness. It achieves this by jointly rewarding both objectives and training a lightweight discriminator in an adversarial loop to measure naturalness. The resulting model generates valid simulated conversations 46.2% of the time, more than triple the success rate of the best prior simulator.

By Mustafa Omer Gul, Philippe Laban, Tanya Goyal, Jennifer Neville
arXiv Computer Vision
Oct 23

MeSS: City Mesh-Guided Outdoor Scene Generation with Cross-View Consistent Diffusion

MeSS (Mesh-based Scene Synthesis) is a method that generates high‑quality, style‑consistent outdoor scenes using city mesh models as geometric priors. The pipeline first creates geometrically consistent sparse views with Cascaded Outpainting ControlNets, then densifies intermediate views via AGInpaint, and finally removes visual inconsistencies with GCAlign. Simultaneously, a 3D Gaussian Splatting scene is reconstructed from the mesh, enabling diverse rendering styles after synthesis.

By Xuyang Chen, Zhijun Zhai, Kaixuan Zhou, Zengmao Wang, Jianan He, Dong Wang, Yanfeng Zhang, Mingwei Sun, Xuqin Wang, R\"udiger Westermann, Tao Wu, Konrad Schindler, Liqiu Meng
arXiv AI
Oct 23

SAKURA: Serving Attention Key-Value State Under Rate Adaptation for Private Image Editing on Phones

SAKURA is a cloud-edge system that delivers reusable attention key‑value state to mobile devices for diffusion‑based image editing while respecting a bandwidth budget. It compresses the state with a token‑importance‑aware method, dynamically selects cache representations based on network conditions, and overlaps transfer with on‑device computation. Experiments on a Snapdragon 8 Elite Gen 5 phone show that SAKURA cuts latency by up to 3.27× and reduces cache storage by 3.06× compared to uncompressed caches, all while maintaining image quality.

By Shuncheng Jie, Desen Sun, Ri Hong, Yuyang Tian, Hong Zhang, Sihang Liu
arXiv AI
Oct 23

Allocating Indivisible Goods to Strategic Agents: Pure Nash Equilibria and Fairness

The paper studies how to allocate indivisible goods to strategic agents with additive valuations without monetary transfers. It investigates whether mechanisms can guarantee pure Nash equilibria while ensuring fairness, focusing on envy‑freeness up to one good (EF1) and envy‑freeness up to any good (EFX). The authors show that the Round‑Robin algorithm yields EF1 allocations at every pure Nash equilibrium, and that a cut‑and‑choose algorithm for two agents produces EFX and maximin share fair allocations in equilibrium, a property absent in the non‑strategic setting.

By Georgios Amanatidis, Georgios Birmpas, Federico Fusco, Philip Lazos, Stefano Leonardi, Rebecca Reiffenh\"auser
arXiv Machine Learning
Oct 23

Probing and Steering for Safety-Relevant Behaviours in Large Language Models

The paper introduces activation probes—directions in a language model’s residual stream—that can both detect and steer safety‑relevant behaviors such as deception, high‑stakes awareness, and sycophancy. By evaluating different estimators (difference‑of‑means vs. logistic regression) and token‑pooling schemes (last, mean, learned attention) on Llama‑3.1‑8B‑Instruct, the authors find that logistic regression excels at detection while difference‑of‑means excels at steering, except for sycophancy where the pattern reverses. The study highlights that the optimal pooling method, layer choice, and steering metric vary by behavior and data, and provides exported probes that enable an interactive interface for scoring and steering text.

By Nimun Bajwa, Massimo Poesio
arXiv Machine Learning
Oct 23

Lean Finder V2: Semantic Search for Mathlib That Disambiguates User Queries

Lean Finder V2 is a semantic search framework for Mathlib that resolves ambiguous user queries by combining adversarial training at the model level with input-level query rewriting. It addresses the common scenario where mathematicians and coding agents issue short, context‑poor queries, which existing search engines struggle to interpret. On the MathlibQR benchmark, Lean Finder V2 achieves at least an 11% relative improvement over prior engines and further boosts retrieval performance through optimized query rewriting, while also enhancing downstream theorem‑proving success when integrated with Claude Code.

By Jialin Lu, Xiaoyu Liu, Hengrui Luo, Rodrigo Stehling, Linxi Liu, Weiran Sun, Yuepeng Wang, Wuyang Chen
arXiv Machine Learning
Oct 23

DISC: Domain-Integrated Similarity Cost for Time-Series Dataset

The paper introduces DISC, a Domain-Integrated Similarity Cost framework that learns to aggregate multiple similarity metrics for time‑series datasets. By calibrating on transfer observations, DISC selects, weights, and combines metrics to better predict downstream forecasting performance than any single metric. The authors demonstrate that task‑dependent similarity estimation improves alignment with actual transfer behavior in out‑of‑distribution and transfer‑learning scenarios.

By Yen-Ku Liu, Hongjie Chen, Franck Dernoncourt, Ryan Rossi
arXiv Machine Learning
Oct 23

TerraTransfer++: Demonstration-Free End-to-End Planner

TerraTransfer++ demonstrates a demonstration‑free, end‑to‑end autonomous driving framework that leverages self‑play in vectorized simulators to pretrain a policy. The method distills self‑play rollouts into a flow‑matching trajectory teacher and trains lightweight adapters on a frozen vision backbone, using action KL divergence, a batch‑relational low‑rank loss, and trajectory‑matching loss. The resulting closed‑loop policy surpasses all published imitation‑based methods on the HUGSim benchmark, while the planner achieves state‑of‑the‑art performance on NAVSIM navhard and navtest without any human demonstrations.

By Zikang Xiong, Weixin Li, Zhouchonghao Wu, Akshay Rangesh, Saarth Bonde, Grantland Hall, Chen Tang, Yihan Hu, Wei Zhan
arXiv Machine Learning
Oct 23

Reliable AI-Assisted Decisions via Selective Auditing

The paper introduces a framework for online monitoring of AI systems that produce decisions with associated confidence scores. It formulates the problem as an online constrained optimization task, aiming to minimize expert auditing while ensuring that the proportion of unaudited AI errors remains below a user‑defined threshold. The authors present a family of algorithms with provable error‑control guarantees, including an optimally efficient method that achieves the best possible auditing rate for both adversarial and i.i.d. data streams, and demonstrate its effectiveness on tasks such as screening harmful requests, text annotation, and data labeling.

By Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani, Tijana Zrnic
arXiv Machine Learning
Oct 23

Bayesian Ambiguity Contraction-based Adaptive Robust Markov Decision Processes for Adversarial Surveillance Missions

The paper introduces an adaptive Robust Markov Decision Process framework for autonomous Intelligence, Surveillance, and Reconnaissance missions using Collaborative Combat Aircraft. It models adversarial uncertainty with finite threat prototypes and refines policies by eliminating inconsistent threat models, enabling a shift from worst‑case protection to more efficient mission strategies while preserving robustness. Experiments demonstrate higher mission rewards and fewer exposure events compared to nominal and static robust planners across various network topologies.

By Jimin Choi, Max Z. Li
arXiv Computation and Language
Oct 23

Spectral Smoothing and Spiking Perturbations Reconfigure Capability Profiles and Selective Task Performance in Large Language Models

The paper investigates how post‑hoc spectral smoothing and spiking interventions on transformer feed‑forward projection matrices affect large language models. By altering singular‑value spectra while preserving singular vectors and the geometric mean, the authors evaluate the impact on a range of tasks—including reasoning, truthfulness, hallucination, safety, and personality probes—across several Llama and Qwen models. The results reveal that structured perturbations can selectively suppress or enhance specific capabilities, suggesting a lightweight method for probing how different abilities are organized and traded off within these models.

By Qince Luan, Jianhui Xu, Yang Xu
arXiv Computation and Language
Oct 23

UniGuard: Multi-View Guardrail Learning for Cross-Interface Safety Generalization

UniGuard introduces a training framework that learns interface‑invariant safety representations by leveraging multiple semantically aligned views of the same safety case. The authors created the UniGuard‑FT dataset, transforming 1,000 public safety cases into prompt, response, multi‑turn conversation, agent‑trace, and multi‑agent‑trace formats, and trained a 0.6B model using classification, supervised contrastive learning, and asymmetric confidence calibration losses. On unseen‑interface benchmarks (ATBench, PolicyGuardBench, DynaBench), UniGuard‑0.6B outperforms its pretrained guardrail by an average of 45.12 F1 points and reduces expected calibration error by 35.41 points, while remaining competitive with much larger guardrails such as Claude‑Sonnet‑4.5 and GPT‑5.2.

By Viet-Thanh Pham, Thuy-Trang Vu, Junae Kim, Rollin Omari, Dinh Phung, Gholamreza Haffari, Lizhen Qu
arXiv Computation and Language
Oct 23

The Geometry of Metaphor: Probing and Steering Metaphor Representations in Large Language Models

The paper investigates how metaphoric concepts are encoded in large language models (LLMs) by applying mechanistic interpretability methods to identify causal subspaces across various architectures and sizes. It demonstrates that manipulating these subspaces allows steering of metaphor-related outputs on both classification and generation tasks, effectively controlling the metaphoricity of LLM responses.

By Shihui Li, Mette Andersen, Martha Lewis
arXiv Computation and Language
Oct 23

The Imperfective Paradox in Large Language Models

The paper examines whether large language models truly understand the compositional semantics of events, focusing on the Imperfective Paradox where past progressive implies event completion for activities but not for accomplishments. It introduces the ImperfectiveNLI diagnostic dataset to test this distinction across semantic classes. Experiments reveal a Teleological Bias in open-weight models, where they hallucinate completion for goal-oriented events and struggle with atelic verbs, suggesting that these models function more as predictive narrative engines than logical reasoners.

By Bolei Ma, Yusuke Miyao
arXiv Computer Vision
Oct 23

GraphCAD: Geometry-Aware Diffusion on Edge-Adjacency Graphs for CAD Generation

GraphCAD introduces a geometry-aware generative framework for Computer-Aided Design (CAD) that models CAD construction using an Edge-Adjacency Graph (EAG), a novel graph-based representation. The method employs a hybrid graph diffusion process to jointly capture discrete topological connectivity and continuous spatial geometry, and incorporates a geometry-aware denoising mechanism that maps node attributes to SE(3)-equivariant encoded point sets. Experiments on the newly created EAGBench dataset demonstrate that GraphCAD can generate complex CAD models with more than 50 faces, achieving state‑of‑the‑art geometric fidelity and distributional alignment compared to sequence‑based baselines.

By Wenhao Zheng, Xianggen Liu, Li Yue, Jiancheng Shi, Jiancheng Lv
arXiv Computer Vision
Oct 23

Where Do Long Captions Fail? Position-Aware Diagnosis and Reinforcement Learning for Detailed Image Captioning

The paper introduces a position‑aware rubric that breaks down long image captions into atomic visual claims, allowing verification of generated text against these claims and revealing distinct failure modes: unsupported specificity in the middle and vague language at the end. Using this diagnosis, the authors propose PAGER, a reinforcement learning objective that provides separate rewards for middle evidence, tail continuation, claim coverage, and length control. Experiments on detailed‑caption benchmarks show that PAGER improves position‑wise grounding and pairwise caption preference while maintaining overall caption quality.

By Yuanze Hu, Zhichao Yang, Junwei Jing, Xin Yang, Wenxuan Liu, Tinghai Zhang, Jia Fu, Hongzhi Zhang, Xiangyu Wu, Ruiming Tang, Tingting Gao, Han Li
arXiv Computer Vision
Oct 23

A Sparse Topometric Graph for Scalable Cross-Session Indoor Navigation

The paper introduces a sparse topometric graph for indoor navigation that encodes local 3D geometry in nodes and distance‑weighted edges for traversable connectivity. A learned graph‑matching network aligns subgraph structures across different sessions, enabling robust point‑goal navigation even with scene changes, sensor variations, or limited overlap. Experiments on Gibson and 3D‑FRONT datasets show higher success rates and a memory reduction of over two orders of magnitude compared to point‑level maps.

By Eun Sun Lee, Minkyun Seo, Gwangtak Bae, Junho Kim, Seunggu Kang, Jaesik Park, Young Min Kim
arXiv Computer Vision
Oct 23

VGGT-Bridge: Beyond Sequential Pose Graphs via Coarse-Stride Skip Edges

VGGT-Bridge extends feed‑forward visual geometry transformers by adding long‑range skip edges between non‑adjacent chunks, reducing drift in multi‑view 3D reconstruction. It processes sparsely sampled coarse chunks to bridge distant fine chunks, corrects first‑frame scale bias with reverse‑direction sampling, and incorporates a loop‑aware policy for compatibility with loop closures. The method achieves up to 28.3% absolute trajectory error reduction on KITTI, 18.8% on Virtual KITTI, and 10.0% on Waymo Open compared to the SwiftVGGT baseline.

By Sungjae Choi, Hanna Bae, Sunghyun Baek, Junmo Kim