AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv AI
Sep 24

Teach-to-Crash: A Closed-Loop Student-Teacher LLM Framework for Collision-Inducing Test Scenario Generation

Teach-to-Crash is a closed‑loop testing framework that uses a dual‑LLM architecture to generate collision‑inducing scenarios for autonomous driving systems. A high‑reasoning Teacher LLM controls the search when collision metrics stagnate, while a low‑reasoning Student LLM produces simulator‑executable scenarios in JSON. In a CARLA case study, Teach‑to‑Crash achieved the highest collision hit rate (90.79 %), the shortest mean time‑to‑collision (18.31 s), and superior diversity and avoidability metrics compared to other methods.

By Zaid Ghazal, Khouloud Gaaloul, Bruce Maxim
arXiv AI
Sep 24

Backdoors Leave Structural Traces: FedMAST for Backdoor Detection and Containment in Federated Learning

The paper introduces FedMAST, a Federated Multi‑Axis Structural Tracing defense designed to detect and contain backdoor attacks in federated learning. FedMAST evaluates client updates through complementary structural, spectral, and historical evidence, applying tiered filtering and round‑level containment. In experiments across six backdoor attacks, FedMAST consistently achieves lower attack success rates while preserving high main‑task accuracy.

By Srinivasan Subramanian, Kazi Aminul Islam, Md. Abdullah Al Hafiz Khan
arXiv AI
Sep 24

FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration

FDE-Bench is a benchmark that tests large language model agents on 136 deployment‑configuration tasks involving Docker, Compose, and Kubernetes, in both greenfield and diagnose‑and‑repair scenarios. Agents submit declarative artifacts that are rebuilt and redeployed in a clean environment, and four binary check layers evaluate build, readiness, behavior, and specification conformance without an LLM judge. The benchmark includes a release gate, detailed check annotations, adversarial strategies, and reports that state‑of‑the‑art models resolve 52.9–75.0 % of tasks, while zero‑intelligence baselines solve none.

By Weihang Ding, Junfei Zhan, Yueting Li, Qirong Guo
arXiv AI
Sep 24

InGuard: Towards Generalized Inner Guardrail for Safe Text-to-Image Generation

InGuard introduces an inner guardrail for text-to-image generation that operates within the model’s own representations, avoiding external classifiers. It grades prompts using the text encoder’s embeddings, modifies risky embeddings with SAGE to produce safe images, and employs a latent detector to halt generation early. Evaluated on the RevGen Safety Benchmark, InGuard achieves a 97.9–98.8% safety rate across five open-weight models while reducing benign disturbances, model parameters, and denoising steps.

By Zeyu Wang, Xiaodan Li, Zhiwen Li, Yuefeng Chen, Hui Xue
arXiv AI
Sep 24

Which Objectives Need a Dial? Predicting Objective Conflict and Covering Trade-offs in Steerable Pluralistic Alignment

The paper investigates how to create steerable AI models that can balance multiple, sometimes conflicting objectives, a necessity for pluralistic alignment. Using Multi-Objective Direct Preference Optimization (MODPO), the authors examine when a single model can improve two objectives simultaneously and how to cover many trade‑offs without training separate models. They find that two pre‑training measurements predict objective alignment for human‑annotated data but not for AI‑annotated data, and that selecting the nearest trained model or merging parameters can broaden trade‑off coverage, though neither approach consistently matches direct training.

By David Tsoi, Esra D\"onmez
arXiv AI
Sep 24

DRSR: Learning Set-Level Deletion Risk for Efficient Long-Horizon Agents

The paper introduces Direct Relational Set‑Risk Pruning (DRSR), a method for compressing the history of long‑horizon language‑model agents by selecting deletion sets based on risk constraints rather than independent unit scores. DRSR builds counterfactual supervision offline, then uses a lightweight scorer to predict set‑level harm during deployment, removing the largest safe set while respecting recency, protocol, and budget limits. Experiments on WorkBuddyBench Full260 and Eval40 show that DRSR improves mean reward from 0.699 to 0.802 and reduces token usage by over 20%, with further analyses highlighting the importance of decision‑conditioned relations, retained context, pair interactions, and abstention.

By Mingxuan Wang, Bo Wang, Fei Luo, Guorun Yao, Chao Ning, Yinglong Guo, Hongyue Chen, Yanbiao Ma, Jungong Han
arXiv Machine Learning
Sep 24

When Adaptation Hurts: Split Sensitivity and Person-Level Negative Transfer in Federated Wearable Onboarding

The paper evaluates six onboarding strategies for federated wearable models on five datasets using a leakage‑controlled protocol that fixes source checkpoints and separates calibration from evaluation. Results show that while average accuracy is high, person‑level performance can drop significantly, with some methods causing negative transfer for certain users. The study highlights that mean accuracy alone is insufficient and provides an auditable benchmark and failure map for future development.

By Rahil Aftab, Vineet Kumar Rakesh, Soumya Mazumdar, Tapas Samanta
arXiv Machine Learning
Sep 24

CAST: Context- and Anomaly Structure-Conditioned Time Series Anomaly Generation

CAST is a framework for generating anomalous time series that addresses the scarcity and heterogeneity of anomaly data. It uses a two‑stage approach: pretraining on abundant normal data to learn system dynamics, then finetuning with anomaly structure representations to capture diverse anomaly morphologies. Experiments on real‑world datasets show that CAST outperforms existing methods in both generation quality and downstream task performance.

By Haochen Zhang, Jie Peng, Songyuan Sui, Yu-Chao Huang, Xiangqi Zhu, Tianlong Chen
arXiv Machine Learning
Sep 24

Quality over Quantity: Semi-Supervised Detection of Illicit Bitcoin Flows via Feature Engineering

The paper presents a semi‑supervised learning framework for detecting illicit Bitcoin flows in Shared Send Mixers, using a large historical dataset of 163 million transactions. It demonstrates that the success of SSL depends on data quality rather than sheer volume, with high‑fidelity features such as KeyLinker address clustering and Shared Send Untangling complexity metrics achieving an F1 score of 0.84 on unlabeled data. The study also shows that common heuristics like One‑Time Change introduce noise, underscoring the importance of smarter feature engineering in blockchain forensics.

By Yekaterina Smolenkova, Nickolay Larionov, Nikolay Ivanov, Yury Yanovich
arXiv Machine Learning
Sep 24

Learning from Failures: Heterogeneous Graph Memory for Small Language Model Tool-Using Agents

The paper introduces FRESH, a Failure-aware Retrieval framework that uses Experience-Structured Heterogeneous graphs to transform past successes and failures into structured external memory for tool‑using agents. By explicitly modeling dependencies among tasks, actions, errors, repairs, and execution conditions, FRESH enables frozen language models to reuse reliable strategies, avoid recurring failures, and make safer decisions in stateful tool interactions. Experiments on τ‑Bench and AppWorld with multiple open‑source models demonstrate that FRESH consistently improves task success and tool‑use reliability compared to no‑memory agents and other memory‑based baselines.

By Jiaxing Li, Lei Song, Rui Dong, Youyong Kong
arXiv AI
Sep 24

Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenon

The paper presents a near-complete, nonasymptotic generalization theory for multilayer neural networks using path regularization, applicable to broad Lipschitz loss functions without requiring bounded loss or extreme network hyperparameters. It provides an explicit upper bound that addresses approximation rates in generalized Barron spaces and demonstrates the double descent phenomenon for ReLU networks. The authors claim near-minimax optimality for regression problems and plan to establish matching lower bounds in future work.

By Hao Yu
arXiv Machine Learning
Sep 24

Artificial intelligence surrogates for treatment effect estimation with before-and-after data

The paper proposes a method for estimating treatment effects using AI-generated predictions as surrogates, applied to paired before-and-after measurements for each treated individual. By comparing AI predictions before and after treatment, the approach can identify the average treatment effect on the treated under certain technical assumptions, even when clinical outcomes are never observed for treated subjects. When assumptions are questionable, the authors introduce prediction‑powered inference that corrects bias with a small set of observed outcomes, and validate the method with synthetic and cardio‑oncology data.

By Frances Dean, Anna Neufeld, Joshua Barrios, Geoffrey H Tison, Ahmed Alaa
arXiv Machine Learning
Sep 24

Reliable Federated TinyML Deployment for IoT Security

The paper explores how to combine Federated Learning with TinyML model compression techniques—such as knowledge distillation, structured pruning, and quantization—to create lightweight, privacy‑preserving intrusion detection systems for IoT devices. It evaluates these strategies within a federated training pipeline and finds that training stability is crucial; a server‑coordinated cosine learning‑rate schedule boosts Attack Recall from 46.7% to 93.85% while still allowing significant model compression and efficient edge deployment.

By Younsoo Park, Seokhyoen Bae, Shasi Kumar Ramachandran Prabhu, Suman Saha, Peilong Li
arXiv Machine Learning
Sep 24

FFM-CP: Cross-Backbone Fusion of Vision-Language Foundation Models for Few-Shot Computational Pathology

The paper introduces FFM-CP, a framework that fuses multiple pathology vision‑language foundation models for few‑shot learning. It aligns heterogeneous representations with an Orthogonal Procrustes transformation, then uses a unified graph to refine support‑image features and class prototypes across backbones. Experiments on six histopathology datasets show that FFM‑CP outperforms the best single adapted model in 50 of 54 few‑shot comparisons.

By Anh-Tien Nguyen, Trung DQ. Dang, Nghiem Tuong Diep, Bui Ngoc Han Nguyen, Tan-Ha Mai, Miriam Cindy Maurer, Phuong Hoa Nguyen, Thi Thuy Uyen Nguyen, Youngjun Park, Daniel Sonntag, Duy Minh Ho Nguyen, Anne-Christin Hauschild
arXiv Machine Learning
Sep 24

"What's That Sound?": A Versatile, Robust, and Lightweight Convolutional Transformer for Environment Sound Recognition

The paper introduces RALCT, a lightweight Convolutional Transformer that combines randomized audio augmentations, MFCCs, and log‑mel spectrograms to extract robust features for environmental sound recognition. With only about 310,000 parameters, RALCT achieves state‑of‑the‑art accuracy—over 93% on UrbanSound8K, peaking at 94.56%—making it suitable for deployment on mobile devices. The authors also develop a mobile app that integrates the model to provide real‑time safety alerts for hearing‑impaired users.

By Julia Huang
arXiv Machine Learning
Sep 24

ChronosAttack: Adversarial Tool Scheduling Attacks on LLM Agents

ChronosAttack is a delay‑only scheduling attack that manipulates the timing of authentic tool responses in large language model agents, without altering the content of those responses. By introducing bounded delays, the attack can reorder identical evidence and shift the agents’ final decisions. Experiments on GPT‑5.6 Sol, Gemini 3.6 Flash, DeepSeek V4 Flash, and Claude Sonnet 4.6 show that the attack can cause strong targeted shifts in some models, large opposite shifts in others, and that a single scheduling inversion can dramatically change outcomes. Defenses based on synchronization and order‑consistency reduce the attacker’s influence over observation order.

By Arash Vashagh
arXiv Machine Learning
Sep 24

Dirichlet Process Mixtures of Trees with Gaussian Process Splits: A Bayesian Nonparametric Framework with Posterior Contraction Rate

arXiv:2609. 27930v1 Announce Type: cross Abstract: We propose a Bayesian nonparametric mixture of regression trees with a Dirichlet process prior over tree-parameter pairs, enabling data-driven selection of ensemble size and unifying CART, BART, random forests, and boosting.

By Subhasish Basak, Anik Roy, Sourabh Bhattacharya
arXiv Machine Learning
Sep 24

ForgetMimic: Motion Unlearning for Reinforcement Learning Humanoid Control

ForgetMimic is a motion-level unlearning method for reinforcement learning-based humanoid control. It selectively degrades performance on a chosen subset of motions while preserving the policy’s effectiveness on the remaining motions. Experiments on Unitree G1 and H2 robots across 12 motions show that the method successfully removes memory of designated motions without affecting other behaviors.

By Xukun Luan, Zhongxiang Lei, Chen Gong, Shaowei Li, Yuanguo Bi, Jinyan Liu
arXiv Machine Learning
Sep 24

Integrated Multivariate Segmentation Tree for Heterogeneous Credit Data Analysis in Small- and Medium-Sized Enterprises

The paper introduces the Integrated Multivariate Segmentation Tree (IMST), a new framework that combines financial data and textual information for credit evaluation of small- and medium-sized enterprises. IMST transforms text into numerical matrices via matrix factorization, selects key financial features with Lasso regression, and builds a multivariate segmentation tree using Gini or entropy with weakest-link pruning. Experiments on 1,428 Chinese SMEs show an 88.9% accuracy, outperforming baseline decision trees, SVMs, and neural networks while offering better interpretability and computational efficiency.

By Lu Han, Xiuying Wang
arXiv Machine Learning
Sep 24

Safe learning-based control via function-based uncertainty quantification

The paper presents a method for safe learning-based control that uses function-based uncertainty quantification. By modeling the unknown function as a random function and generating independent realizations, the authors construct uncertainty tubes via the scenario approach that hold with high probability. These tubes, which rely only on sampled realizations, can handle discontinuities and are integrated into a safe Bayesian optimization algorithm to tune control parameters on a real Furuta pendulum.

By Abdullah Tokmak, Toni Karvonen, Thomas B. Sch\"on, Dominik Baumann