AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Machine Learning
5d ago

Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models

The paper presents a recovery‑based shielding framework that guarantees a provable safety lower bound for reinforcement learning in unknown, non‑linear continuous dynamical systems. It combines a backup policy (shield) with the RL agent and uses Gaussian process models to quantify uncertainty and predict safety violations, enabling dynamic recovery to safe trajectories only when needed. The shielded agent’s experience trains the GP models, and policy optimization is performed via internal model‑based sampling, allowing unrestricted exploration and sample‑efficient learning while maintaining strict safety compliance in continuous control tasks.

By Alexander W. Goodall, Francesco Belardinelli
arXiv Machine Learning
5d ago

Repair Brain Damage: Real-Numbered Error Correction Code for Neural Network

The paper introduces a real-valued error correction coding framework for neural networks that detects and corrects both memory faults and computational errors. It imposes structured linear constraints on network weights, allowing efficient error detection and recovery without adding parameters or hurting classification performance. The method uses linear programming and sparse recovery to keep error correction low‑complexity, and is claimed to be the first real‑number‑based ECC scheme addressing both error types simultaneously.

By Ziqing Lu, Myung Cho, Qiutong Jin, Weiyu Xu
arXiv Computation and Language
5d ago

Ctrl-G: Controlling Structural and Interaction Exposure in Knowledge Graph Augmented LLM Reasoning

Ctrl‑G is a knowledge‑graph augmented retrieval‑augmented generation (KG‑RAG) framework that jointly limits structural and interaction exposure when sending graph evidence to a remote large language model (LLM). It constructs query‑conditioned sanitized views via structural aggregation and pruning, and uses a hierarchical controller to prune exploration paths while verifying evidence against the original KG. By reusing abstract, verified reasoning traces in local memory, Ctrl‑G reduces repeated disclosure across questions and achieves the highest accuracy on five of six benchmarks, outperforming the strongest baseline by up to 13.8 points.

By Xingyu Tan, Xiaoyang Wang, Qing Liu, Xiwei Xu, Xin Yuan, Liming Zhu, Wenjie Zhang
arXiv Computer Vision
5d ago

WildOS: Open-Vocabulary Object Search in the Wild

WildOS is a unified system that enables long‑range, open‑vocabulary object search in unstructured outdoor environments by combining safe geometric exploration with semantic visual reasoning. It constructs a sparse navigation graph for spatial memory and uses the ExploRFM vision module to score frontier nodes, predicting traversability, visual frontiers, and object similarity in real time. A particle‑filter localization method estimates distant goal positions, allowing the robot to plan toward far‑away targets, and field experiments show WildOS outperforms purely geometric or vision‑based baselines in efficiency and autonomy.

By Hardik Shah, Manthan Patel, Erica Tevere, Deegan Atha, Marcel Kaufmann, Shehryar Khattak, Marco Hutter, Jonas Frey, Patrick Spieler
arXiv Computer Vision
5d ago

PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts

PixelMoE introduces a pixel‑space generative framework that applies LLM‑style Mixture‑of‑Experts (MoE) to diffusion models, enabling sparse scaling of model capacity without diffusion‑specific routing. It combines Token‑Choice routing, fine‑grained routed experts, a shared expert, and bias‑based load balancing, and adds Pixel‑Space Hierarchical Flow Matching to vary spatial resolution during denoising. Across model scales, PixelMoE outperforms dense counterparts and achieves competitive FID scores on class‑conditional ImageNet‑1K, with code and pretrained models publicly released.

By Ruixiao Dong, Jianmin Bao, Bozhi Luan, Zijie Meng, Xin Tao, Pengfei Wan, Kun Gai, Li Li, Houqiang Li
arXiv Machine Learning
5d ago

Differentially Private Synthetic Data from Sliced Chebyshev Moments

The paper introduces SPLICE, a differentially private synthetic data algorithm that uses noisy sliced Chebyshev polynomial moments to fit a distribution. SPLICE achieves a Sliced Wasserstein-1 distance of ∼O(1/√n) from the target distribution, a bound that does not depend on the data dimension. The authors also prove a new moment‑matching result and provide a practical implementation that outperforms existing DP synthetic data methods on a large benchmark.

By Lucas Rosenblatt, R. Teal Witter, Christopher Musco
arXiv Computation and Language
5d ago

Language Models Show Conformity Bias in RAG: When the Majority Endorses a Poisoned Claim

Retrieval‑augmented generation (RAG) can expose language models to misleading evidence. In a study of 200 SciFact questions, the Gemma‑3‑12b model’s ability to detect poisoned claims dropped from 25% when a single poisoned document was present to 10% when six mutually agreeing poisoned documents dominated the context. The majority condition caused 34 of 38 questions to shift from detected to undetected, indicating a conformity‑like bias.

By Mehrdad Ghassabi, Audrina Ebrahimi, Zahra Zojaji, Amin Milani Fard
arXiv Machine Learning
5d ago

Cyclical Entropy Eruption: Entropy Dynamics in Agent Reinforcement Learning

The paper introduces the phenomenon of cyclical entropy eruption in agent reinforcement learning, where entropy rises sharply and then subsides in recurring cycles, unlike the steady low entropy seen in single‑turn reasoning RL. It breaks this dynamic into three phases, offering theoretical and empirical insights into the mechanisms driving the oscillation. The authors propose SEAL, a lightweight auxiliary loss that separates correct and incorrect trajectories in representation space, which stabilizes training and improves downstream agent performance across various benchmarks and algorithms.

By Wendi Li, Shawn Im, Sharon Li
arXiv Computation and Language
5d ago

CoFiBreak: A Coarse-to-Fine Diversity Reward for Effective Multi-Turn Jailbreak

CoFiBreak introduces a coarse‑to‑fine diversity reward for Group Relative Policy Optimization (GRPO) that enhances the variety of attack strategies in automated red‑teaming of large language models while maintaining high attack success and low inference cost. The coarse reward encourages underrepresented strategies within each GRPO group, and the fine reward discourages reuse of personas, framings, and output formats after harmful‑intent information is removed from prompt embeddings. Trained against Llama3.1‑8B‑Instruct, CoFiBreak achieves 1.6–2.0× more distinct successful strategies than the strongest baseline, attains 98.8% ASR@10 on AdvBench, and transfers effectively to unseen frontier LLMs such as Claude‑Opus‑4.6.

By Jialin Song, Xiaodong Liu, Weiwei Yang, Mingqian Feng, Chengquan Guo, Qiyao Ma, Carolyn Buractaon, Anson Ho, Wuyang Chen, Christopher White
arXiv Computation and Language
5d ago

Aligning MLLMs with the Latent Structure of Human Cognition via Behavior-Derived Semantic Dimensions

The paper introduces a framework that aligns multimodal large language models (MLLMs) with human cognitive structure by leveraging a behavior-derived semantic embedding space built from 4.70 million human Odd-One-Out judgments. It maps each triplet to the most salient decision dimension and generates dimension‑grounded linguistic rationales, enabling MLLMs to learn choices and justifications consistent with human criteria. Experiments show improved consistency with human similarity judgments, competitive benchmark performance, and increased representational alignment with human fMRI data.

By Ning E, Changde Du, Yizhuo Lu, Huiguang He
arXiv Computation and Language
5d ago

TRIDENT and SHAKTI: A Framework and Benchmark for Hindi AI-Text Detection

The paper introduces SHAKTI, a large-scale Hindi benchmark for detecting AI-generated text, covering multiple genres, LLM families, prompting strategies, and adversarial transformations. It evaluates the benchmark with human annotation and examines linguistic, grammatical, and sentiment differences between human and AI texts. Building on these insights, the authors propose TRIDENT, a round‑trip translation‑based detection framework with three variants that outperforms existing baselines and demonstrates robustness across seven languages and adversarial settings.

By Shifali Agrahari, Sujit Kumar, Sanasam Ranbir Singh