AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,884 stories · RSS feed

arXiv Machine Learning
Sep 3

DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Driving

DiDrive introduces a risk‑aware hierarchical diffusion framework for offline reinforcement learning in autonomous driving. It combines a low‑level risk‑gated encoder with a high‑level contextual modulator to filter redundant state information, and a 3DICE policy optimization that reduces out‑of‑distribution overestimation and stabilizes gradients. On the CARLA benchmark, DiDrive outperforms baselines such as IQL, CQL, and Diffusion‑QL, achieving an 85% success rate and a 4295.68 average reward in dense traffic with 60 vehicles.

By Qisong Guo, Jingtang Chen, Zhilin Chen, Pei Xu, Mingjian Fu, Wenxi Liu, Yuanlong Yu
arXiv Computation and Language
Sep 3

Selective Knowledge Edit Reversal via Gated Singular Vector Shrinkage

The paper introduces a spectral-based method for selectively reversing edits in large language models. It targets specific edited facts while preserving other modifications by identifying edit-sensitive components in the dominant singular subspace of the edited weights. Experiments show the approach effectively removes unwanted edits without erasing beneficial ones.

By Weifeng Jiang, Ruirui Chen, Qianren Mao, Junnan Liu, Qili Zhang, Kwok-Yan Lam
arXiv Machine Learning
Sep 3

Private Computation Space: Experience with Trusted Multi-Cluster Federated Learning for Agriculture

The paper introduces the Private Computation Space (PCS), an open‑source federated learning system designed for agriculture that protects farmer data using asynchronous federated learning, differential privacy, and trusted execution environments. PCS runs on commodity hardware and is resilient to rural infrastructure challenges. In two real‑world deployments—nitrogen monitoring in New York and evapotranspiration prediction in California—PCS achieved a Dice Similarity Coefficient of 0.71 and an $R^2$ of 0.84, improving single‑site model accuracy by 22.4% and 9.1% respectively while preserving privacy.

By Shuangyu Lei, Muhammad Salman Abid, Jacob Belding, Sam Mosher, Manushi B. Trivedi, Shivranjani Baruah, Liam Wickes-Do, Andrew Anderson, Braulio Dumba, Alyssa Whitcraft, Ritvik Sahajpal, Sijin Li, Kelly Robbins, Michael Gore, Margaret Frank, Steven Wolf, Liz Jones, Abraham Stroock, Kaitlin Gold, Hakim Weatherspoon
arXiv Machine Learning
Sep 3

Context Inference Attacks Without Jailbreaks

The paper investigates privacy risks in agentic AI systems that assemble sensitive data into a hidden context before responding. It introduces context‑inference attacks, a security game that evaluates how well attackers can recover this hidden context under varying levels of knowledge and indirect delivery. Experiments show that even with controls such as instructions not to disclose, logit suppression, and context dilution, agents can leak significant contextual information, achieving high success rates across multiple attack settings.

By Prince Jha, Samuele Poppi, Nils Lukas
arXiv Machine Learning
Sep 3

GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models

The paper introduces GPTBIAS, a framework that uses powerful large language models like GPT‑4 to evaluate bias in other LLMs. It employs specially crafted prompts called Bias Attack Instructions to probe for bias and outputs a bias score along with detailed information such as bias types, affected demographics, keywords, reasons, and improvement suggestions. Extensive experiments demonstrate the framework’s effectiveness and usability.

By Jiaxu Zhao, Meng Fang, Shirui Pan, Wenpeng Yin, Mykola Pechenizkiy
arXiv Machine Learning
Sep 3

GONE: Structural Knowledge Unlearning via Neighborhood-Expanded Distribution Shaping

The paper introduces GONE, a benchmark for evaluating knowledge unlearning in large language models using structured knowledge graphs, and presents Neighborhood-Expanded Distribution Shaping (NEDS), a framework that leverages graph connectivity to separate forgotten facts from their semantic neighborhood. GONE disentangles direct fact removal, reasoning-based leakage, and catastrophic forgetting, while NEDS achieves high unlearning efficacy and locality on LLaMA-3-8B and Mistral-7B. The dataset is publicly available on Hugging Face.

By Chahana Dahal, Ashutosh Balasubramaniam, Zuobin Xiong
arXiv Computation and Language
Sep 3

SpeakPay: Domain-Adaptive LoRA Fine-Tuning of Whisper for Low-Resource Nepali Financial Speech Recognition

SpeakPay is a voice‑first digital wallet designed to make mobile payment apps in Nepal accessible to visually impaired users. The paper introduces NepFinSpeech‑403, a 403‑utterance Nepali financial voice command dataset, and demonstrates that fine‑tuning Whisper large‑v2 with LoRA reduces the Word Error Rate from 129.95% to 42.58% and improves Devanagari numeral recognition from 0.0% to 73.9%. Domain adaptation also boosts the Transaction Success Rate from 1.67% to 33.33%, with as few as 100 domain‑specific utterances halving the zero‑shot WER.

By Biraj Subedi
arXiv Computation and Language
Sep 3

Candidate Generation and Definition-Guided Verification for Sentence-Level Depression Symptom Recognition

The paper introduces a two‑stage framework for recognizing depression symptoms at the sentence level. First, a contrastively fine‑tuned sentence encoder generates a symptom candidate for each sentence. Then, a fine‑tuned language model verifies the candidate’s presence or absence by comparing the sentence, its context, and a diagnostic definition, ensuring the model’s judgment aligns with that definition before responding.

By Weiming Li, Catarina Barata, Miguel Constante, Joao Sanches
arXiv Computation and Language
Sep 3

A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models

The paper presents a tri‑agent framework for evaluating large language models’ question‑clarification abilities. It involves a Question Clarifying Agent that identifies ambiguities and asks follow‑up questions, a Respondent Agent that simulates human replies, and an Evaluator Agent that judges the dialogue using metrics such as ambiguity handling, question quality, dialogue efficiency, language appropriateness, and intent alignment. The authors illustrate the approach with synthetic supply‑chain data and discuss validating the evaluator against human judgments.

By Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra
arXiv Computation and Language
Sep 3

Do Cantonese-Adapted Language Models Better Predict Cantonese Reading? A Cross-Model Eye-Tracking Evaluation

The study investigates whether language models trained specifically on Cantonese better predict human reading patterns by comparing eye-tracking data with information-theoretic metrics derived from several models. Two adaptation contrasts were examined: a lightly adapted CKIP GPT-2 Tiny versus its Cantonese derivative JED351, and a heavily adapted Qwen2.5-7B versus CantoneseLLM-7B. Results show that the extensively Cantonese-trained CantoneseLLM-7B consistently outperforms others on lexical surprisal and joint metrics, while entropy reduction favors the less adapted CKIP model, indicating that deeper Cantonese-specific training can improve predictive alignment but that rankings vary by metric.

By Ziqi Zhang, Emmanuele Chersoni, Mohammad Momenian
arXiv Machine Learning
Sep 3

Poisoning Attacks on the PGM-index

The paper investigates the vulnerability of the PGM-index to poisoning attacks, introducing PGM-attack—a method that sequentially inserts adversarial keys to inflate the number of segments in the index’s piecewise linear approximation. Experiments show that poisoning just 10% of keys can increase segment count—and thus index size—by up to 120×, with the attack achieving at least 52% of the theoretical optimum. The attack also transfers to other learned indexes, especially those based on PLA, highlighting an intrinsic weakness in the PGM-index’s optimization objective.

By Atsuki Sato, Martin Aum\"uller, Yusuke Matsui
arXiv Computation and Language
Sep 3

User Feedback Provides a Unique Signal that LLMs Can not Detect

The paper argues that user feedback from real interactions is a valuable learning signal for Large Language Models (LLMs), contrary to recent claims that it is too noisy to use. By creating synthetic data with a clear ground truth and testing on naturalistic data, the authors show that revisions guided by user feedback fix targeted issues more often than baseline revisions. They further reveal that current evaluation methods bias against feedback‑driven improvements, as judges tend to overlook genuinely corrected responses and favor inferior baselines.

By Shachar Don-Yehiya, Leshem Choshen, Omri Abend
arXiv Computation and Language
Sep 3

Privacy Washing: Detecting Internal Contradictions in Privacy Policies

The study introduces a four‑stage pipeline to detect internal contradictions—termed privacy washing—in privacy policies. Applied to two corpora (123 policies from 2026 and 115 from 2015), the pipeline identifies contradictions in 12.2% of the newer policies and 36.5% of the older ones, with third‑party sharing conflicts being the most common. A stability re‑run confirms similar prevalence rates and shows that the majority of contradictions are consistent across different model configurations.

By Thomas Brackin
arXiv Computation and Language
Sep 3

HarmReduction: Benchmarking LLMs in Harm Reduction Information Provision to Support People Who Use Drugs

The paper introduces HarmReduction, a benchmark for evaluating large language models (LLMs) on their ability to provide accurate and safe harm reduction information to people who use drugs (PWUD). The benchmark, HR-Basic, contains 2,160 question‑answer‑evidence pairs covering safety boundary checks, quantitative value provision, and polysubstance risk inference. Experiments show that even state‑of‑the‑art LLMs struggle with accuracy and can pose severe safety risks, underscoring the need for a dedicated evaluation framework.

By Kaixuan Wang, Chenxin Diao, Jason T. Jacques, Zhongliang Guo, Shuai Zhao
arXiv Computation and Language
Sep 3

SHARD: Safe and Helpful Alignment via Self-Reframing Distillation

SHARD is a self‑reframing distillation technique designed to enhance the safe‑helpfulness of large language models. It rewrites sensitive prompts to reveal benign intent, reframes the model’s original responses into safer, more helpful versions, and then fine‑tunes the model on these self‑reframed outputs. Experiments on DNA and the English subset of LINGUASAFE show that SHARD improves helpfulness across various model families while maintaining safety, and performs competitively with distillation from larger teacher models.

By Viswonathan Manoranjan, Amogh Gupta, Anvesh Rao Vijjini, Thomas Hofweber, Snigdha Chaturvedi
arXiv AI
Sep 3

From High-Dimensional Spaces to Verifiable ODD Coverage for Safety-Critical AI-based Systems

The paper proposes a structured method for verifying Operational Design Domain (ODD) coverage in safety‑critical AI systems, particularly for aviation. It combines parameter discretization, constraint‑based filtering, and criticality‑based dimension reduction to create a multi‑step verification process. Using simulation data from AI‑based mid‑air collision avoidance research, the authors demonstrate how this approach can meet EASA’s requirement for complete ODD coverage in high‑dimensional spaces.

By Thomas Stefani, Johann Maximilian Christensen, Elena Hoemann, Frank K\"oster, Sven Hallerbach
arXiv AI
Sep 3

PIE-APT: Abductive Planning over Temporal Dynamic Knowledge Graphs via Incremental Reasoning

PIE-APT introduces a unified framework for abductive planning over Temporal Dynamic Knowledge Graphs (TDKGs) using two modules: PIE-Abducer, which performs incremental direct-derivation abduction, and PIE-APT, which interleaves backward‑chaining A* search with PIE-Abducer to generate action sequences and abductive assumptions. The approach operates natively on the expressive SROIQ Description Logic, leveraging an incremental reasoner to maintain decidability and bypass the Ramification Problem. Evaluation on four OWL benchmarks demonstrates qualitative superiority over classical planners and shows that the direct‑derivation method outperforms a Minimal Hitting Set baseline in abductive enrichment.

By Amir Hossein Sharafi, Alireza Shahbazi
arXiv AI
Sep 3

Automated Researchers Can Mitigate Well-characterized Alignment Failures

The paper investigates whether automated alignment researchers (AARs) can post‑train language models to reduce well‑characterized alignment failures such as deception, sycophancy, and jailbreaks while preserving general capability. Across ten failures, the strongest AAR methods significantly lower targeted failures and generalize to held‑out benchmarks, larger models, and multi‑turn audits. In contrast, a human baseline of 28 experienced researchers, given eight hours to devise one‑shot methods, underperformed the best AAR approaches, and providing human ideas to AARs did not improve results.

By Chen Yueh-Han, Jiaxin Wen, Jan Hendrik Kirchner
arXiv AI
Sep 3

Inference-Time Optimization of Prompt Embeddings in Diffusion Models: A Comparison of sep-CMA-ES and Adam

The paper investigates inference‑time optimization of prompt embeddings for the Stable Diffusion XL Turbo model, comparing the gradient‑free Separable Covariance Matrix Adaptation Evolution Strategy (sep‑CMA‑ES) with the gradient‑based Adam optimizer. Using a weighted objective that blends LAION Aesthetic Predictor V2 and CLIPScore, the study evaluates 36 prompts under three weighting regimes (aesthetics‑only, balanced, alignment‑only). Across all settings, sep‑CMA‑ES consistently outperforms Adam in objective value, while also demonstrating favorable divergence metrics and lower compute and memory footprints, indicating its effectiveness as an inference‑time optimizer without requiring model fine‑tuning.

By Dom\'icio Pereira Neto, Jo\~ao Correia, Penousal Machado
arXiv Machine Learning
Sep 3

Source-Free Class Relearning: Diagnosing Forgetting in Class Unlearning

The paper investigates whether a model that has undergone class unlearning can still recover forgotten classes without access to original data. It introduces a white‑box audit method that generates synthetic probes in representation space, filters them by confidence, and relabels boundary‑adjacent probes as the forgotten class. The authors define a Relearning Score to quantify recovery while preserving retain performance, and demonstrate that several unlearning techniques on CIFAR‑10, CIFAR‑100, and TinyImageNet can be fully recovered in a source‑free setting, sometimes even outperforming a retrained reference.

By Zahra Dehghani, Pablo Piantanida, Mohammadhadi Shateri