AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv Machine Learning
Oct 8

Brain alignment of reasoning and action representations from vision-language and action models during naturalistic gameplay

arXiv:2605.19352v2 Announce Type: replace-cross Abstract: Understanding how humans and artificial intelligence systems predict and plan by interacting with their environment is a fundamental challeng...

By Subba Reddy Oota, Anant Khandelwal, Khushbu Pahwa, Satya Sai Srinath Namburi, Tanmoy Chakraborty, Bapi S. Raju, Manish Gupta
arXiv Computation and Language
Oct 8

ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation

arXiv:2610.09163v1 Announce Type: new Abstract: Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when...

By Arkajyoti Chakraborty, Aryan Tayal, Ishika Agarwal, Tanner Sorensen, Justin Chiu, Alessandro Di Bari, Neha Gupta, Andreas Stolcke
arXiv Machine Learning
Oct 8

Estimating Uncoded Crash Factors with Tabular Foundation and System One Models: Kumo Tabular and Jev

The paper presents a system that combines coded police crash records with officer narratives to estimate uncoded crash factors in Texas from 2017‑2025. It uses an in‑context tabular foundation model (Kumo Tabular) to process coded data, a calibrated System One model (Jev) to read sampled narratives, and human judgments to recalibrate probabilities, producing validated population estimates for factors such as hydroplaning, medical episodes, fatigue, animals, and phone use. The approach demonstrates that narrative analysis uncovers more injury crashes than coded fields and that a single additional round of human review can significantly tighten estimate precision. "whyItMatters":"The system provides safety offices with more accurate counts, a discordance map, a validated re‑read list, and a reading budget, enabling better resource allocation for road‑safety interventions."

By Amir Rafe, Subasish Das
arXiv Machine Learning
Oct 8

CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning

CM-DPO introduces a Constraint-Margin Direct Preference Optimization method that replaces the binary preference signal of traditional DPO with a continuous margin based on a deterministic symbolic verifier, scaled by violation severity. Hard and soft constraints are separated via a lexicographic objective to prevent hard constraints from being traded off against preferences. Using procedurally generated constraint profiles and minimal-edit distillation within the SynPlan-R framework, an 8B model fine‑tuned with CM‑DPO achieves high pass and solve rates on several planning benchmarks, matching multi‑agent systems with lower latency and outperforming GPT‑4o on unseen tasks.

By Rabimba Karanjai, Qun Gu, Hemanth Hegadehalli Madhavarao, Wenhuan Sun, Xiaojiao Yu, Suryabhan Singh Hada, Libin N. George, Uma Kona, Richard Williamson, Linsey Pang, Prakhar Mehrotra
arXiv Machine Learning
Oct 8

How Fragile Is On-Device Language Model Safety? Localizing Safety-Critical Parameters for Sparse Fault Analysis

The paper examines the safety of small language models (SLMs) deployed on resource‑constrained, on‑device platforms, focusing on the LLaMA‑2‑7B‑Chat model. It shows that safety‑sensitive behavior is concentrated in a sparse subset of parameters, particularly the MLP down_proj layer, and that altering only 0.19% of these weights can significantly affect safety metrics while leaving overall accuracy largely unchanged. The study suggests that targeted fault analysis and selective integrity protection could improve safety for on‑device, agentic language models.

By Muhammad Zeeshan Karamat, Christiana Chamon Garcia
arXiv Machine Learning
Oct 8

Safe on Average, Unsafe in the Tail: When Is the Episodic-Cost Tail Controllable?

The paper investigates safe reinforcement learning by focusing on the episodic-cost tail rather than just the mean cost. It introduces the use of γαρα_{0.1} (the average cost of the worst 10% of episodes) to classify policies as tail-safe. The authors evaluate standard algorithms on Safety-Gymnasium tasks, identify tail-unsafe policies, and explore constraint families to control tail violations while maintaining return.

By Samuel Tetteh, Cody Fleming
arXiv Machine Learning
Oct 8

Few Bits, One Law: Toward W2A4KV2

The paper introduces CanonQ, a unified quantization‑aware training framework that separates source canonicalization from task‑aware adaptation to enable extreme low‑bit LLM compression. By applying fixed rotations and energy normalization, CanonQ maps heterogeneous tensors to canonical coordinates, allowing frozen Gaussian‑reference codebooks to be reused across layers and models. Joint training then adapts the network to the coupled errors of weight, activation, and KV cache quantization, achieving significant performance gains—up to 14.28× lower perplexity and 57.9% higher zero‑shot accuracy on LLaMA models, and substantial improvements on instruction‑tuned MobileLLM‑Pro‑1B. "whyItMatters":"The framework demonstrates that carefully designed canonicalization and joint training can dramatically improve the efficiency and accuracy of low‑bit quantized large language models, offering a practical path to deploy powerful LLMs with reduced computational and memory footprints."

By Kai Yi, Tarek Elgamal, Sruthikesh Surineni, Vignesh Vivekraja, Soumyadeep Ghosh, Steven Li
arXiv Machine Learning
Oct 8

Global Average Precision for Representation Learning

arXiv:2610.09863v1 Announce Type: cross Abstract: Standard information retrieval metrics, such as mean Average Precision (mAP), assess performance one query at a time, based on how the similarities b...

By Bill Psomas, Mohammad Mahdi, Michalis Thomas, Danda Pani Paudel, Giorgos Tolias, Giorgos Kordopatis-Zilos