AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv Machine Learning
Oct 9

FAITH: Feasibility-Aware Safety-Filtered RL for High-Dimensional Systems

FAITH is a model‑free reinforcement learning framework that separates safety from task performance by using a learned safety filter. The filter approximates the optimal state‑action safety value with a feedforward network, allowing the task policy to optimize return through filtered dynamics without a competing safety term. Experiments on a double integrator, Safety Gym, a 29‑DoF humanoid, and a real Unitree G1 show that FAITH achieves high returns while maintaining near‑perfect safety and handling infeasible starts gracefully.

By Songyuan Zhang, Baljeet Singh, Sarthak Ranjeet Kaingade, Chuchu Fan, Bryan Trinh
arXiv Machine Learning
Oct 9

Brain foundation model-guided source-selective domain adaptation for cross-subject EEG decoding

The paper introduces BFM-MSDA, a brain foundation model-guided multi-source domain adaptation framework for cross-subject motor‑imagery EEG decoding. It first uses pretrained brain foundation model representations to estimate source–target compatibility and select relevant source subjects. A relevance-weighted dual alignment strategy then aligns marginal and conditional feature distributions, achieving higher accuracies on two public MI‑EEG benchmarks and demonstrating the benefits of informed source selection.

By Jinzhou Wu, Baoping Tang, Qikang Li, Yi Wang, Cheng Li, Shujian Yu
arXiv Computation and Language
Oct 9

Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentation

The study investigates how subtle changes in the presentation of a structured judging request can influence the behavior of an answer‑grading model. By adding a single colon to a candidate answer, the researchers found that the model’s false‑acceptance rate rose dramatically—from 1–3% to over 26%—across two benchmark datasets (DROP and GSM8K) and different grading configurations. The effect varied with the presentation format, showing that logically equivalent requests can lead to markedly different vulnerabilities.

By Hongliang Liu
arXiv Computer Vision
Oct 9

Relative Patch Response Learning for Generalizable AI-Generated Image Detection

The paper introduces Relative Patch Response Learning (PRL) to detect AI‑generated images by comparing patch responses across two mixed views of aligned real‑generated image pairs. PRL uses a relative response objective, a reference coherence objective, and an area ranking objective to address contextual shifts caused by self‑attention, providing precise supervision for patch‑level detection. Experiments show PRL outperforms prior methods by 4.3% and 5.9% in average balanced accuracy on standard and in‑the‑wild benchmarks.

By Tianyu Wang, Ouxiang Li, Yanbin Hao, Zhenhua Tang, Shuo Wang
arXiv Computer Vision
Oct 9

OPERA: Object Perception Enhances Single-view 3D Reconstruction

OPERA is a framework that improves single‑view 3D reconstruction by guiding multi‑view diffusion sampling with pretrained perception models through lightweight alignment modules. The modules are trained independently while keeping the generative and perception models frozen, enabling multiple signals to be combined at inference without joint fusion training. Experiments on Google Scanned Objects and OmniObject3D show that OPERA reduces mean Chamfer Distance by 30.4% and 24.8% on the primary baseline compared to unguided reconstruction.

By Y Huynh, Duc Thanh Nguyen, Mohamed Abdelrazek
arXiv AI
Oct 9

Social Pain Disrupts Emotion-Action Brain-State Dynamics in Adolescents with Non-Suicidal Self-Injury

The study examined 106 depressed adolescents, 67 with non‑suicidal self‑injury (NSSI) and 39 without, using EEG microstate analysis during social pain, physical pain, and rest. A deep sequence model distinguished NSSI participants from controls with 68.55% accuracy, revealing weakened transitions between microstates linked to emotion/interoception (MS3) and action preparation (MS5) during social pain. Time‑resolved analyses showed altered recruitment patterns of these microstates in NSSI adolescents, indicating disrupted emotion‑action coupling.

By Ying Xu, Xiaojun Liang, Li Zhang, Yixuan Yuan, Gan Huang, Yongjie Zhou, Zhen Liang
arXiv AI
Oct 9

What to Admit and How to Present: Governing Persistent Memory in LLM Agents

The paper introduces two inference‑time designs—factor‑compiled admission (FC) and permission‑semantic admission (PS)—to govern persistent memory in large language model agents without retraining. FC evaluates entire memory entries, while PS decomposes entries into typed units, both using deterministic policies to decide what recalled information enters the working context and how it is expressed. Experiments on a development suite and an external benchmark show that FC and PS reduce failure rates and cross‑domain leakage compared to verbatim injection, though they increase personalization failures and do not meet all preregistered criteria.

By Chang Liu, Deliang Ding
arXiv AI
Oct 9

Learning Probabilistic Logic Programs with Functional Gradient Guided Language Models

The paper introduces grasp, a neurosymbolic framework that learns probabilistic logic programs by combining functional gradient boosting with large language models (LLMs). In grasp, the weak learner is a first‑order rule and the combinatorial search for new rules is delegated to an LLM proposal oracle, allowing systematic inductive reasoning. Experiments on four relational benchmarks—including molecular toxicity prediction, mutagenesis, and citation matching—show that grasp outperforms purely symbolic, neural, and LLM‑based baselines while producing interpretable weighted rule ensembles.

By Saurabh Mathur, Sahil Sidheekh, Bhavan Vasu, Farbod Tavakkoli, Prasad Tadepalli, Kristian Kersting, Sriraam Natarajan
arXiv AI
Oct 9

An Interpretable Approach to PDE Solution Discovery via Structural Experience Distillation

The paper introduces SED-MCTS, a Monte Carlo tree search method that enhances PDE solution discovery by distilling structural experience from evaluated expressions. Unlike previous approaches that rely on a single terminal score, SED-MCTS uses counterfactual subtree interventions to estimate local structural contributions, guiding the search toward reliable construction edges and preserving useful components in a structural archive. The method extends naturally to coupled multiphysics systems and demonstrates improved performance, efficiency, and robustness across diverse PDE benchmarks, especially under noisy or scarce observations.

By Yunpeng Gong, Huolong Wu, Can Yang, Min Jiang
arXiv AI
Oct 9

Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark

The paper investigates whether the HELM Safety benchmark’s HarmBench dataset measures a single attribute—harmful refusal—using psychometric methods. Multidimensional item response theory indicates HarmBench does not capture a single attribute, and differential item functioning reveals that models from different developers score items differently, suggesting domain‑specific effects. The authors conclude that safety scores should only be used for attribute comparison after confirming single‑attribute validity, warning that aggregated scores can mask saturation and conflated behaviors.

By Christopher M. Stewart, Preston Botter, Natalie Sarabosing, Muye Zhang, Rachel Phinnemore, Shalini Ghosh, Hong Shen, Hoda Heidari
arXiv AI
Oct 9

From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage

The paper introduces ALERT‑BENCH, a benchmark that simulates enterprise security telemetry and evaluates how large‑language‑model agents triage alerts. Five common reasoning strategies—single‑pass tool use, iterative retrieval, sampled investigations, self‑review, and explicit verification—were tested on 1,247 alerts from a multi‑stage attack scenario, each missing at least 40.4% of attack‑related alerts. Building on these findings, the authors propose AIDA, a multi‑agent framework that enforces explicit decision proposals, independent challenges, and stronger evidence requirements, achieving an F1 score of 0.958 and reducing false‑negative alerts from 40.4% to 3.1% while escalating 18.4% to analysts.

By Saimon Amanuel Tsegai (Daphne), Alex Kantchelian (Daphne), Danfeng (Daphne), Yao, Peng Gao
arXiv AI
Oct 9

Coverage-Aware Reasoning with Medical Tokens for Diagnosis Prediction

The paper introduces CARing, a framework that improves next‑visit diagnosis prediction by representing diagnoses with compositional Semantic IDs (SIDs) and optimizing multi‑label coverage through reinforcement learning. CARing encodes ontology‑enriched disease semantics into compact SIDs, aligns them with natural language and EHR contexts, and employs a coverage reward to encourage diverse diagnosis predictions. On MIMIC‑III and MIMIC‑IV datasets, CARing outperforms all EHR‑trained baselines in weighted F1 and achieves the highest top‑k recall, including R@30 scores above 46% in reasoning mode.

By Kaisong Zhang, Haotian Fang, Junmeng Zhou, Hang Lv, Yulan Pan, Yanchao Tan
arXiv AI
Oct 9

Uncovering and Fixing Collider Bias in Bayesian PINNs

The paper investigates a systematic bias in Bayesian physics-informed neural networks (B-PINNs) caused by their typical collider structure, where independent physical and trajectory parameters become coupled through likelihoods on differential-equation residuals. The authors demonstrate that this modeling choice can drive the posterior away from true parameter values, even with a well-centered prior. They propose a hierarchical chain model that avoids this bias, though it introduces a doubly intractable inference problem, which they solve by discretizing the dynamics and applying particle MCMC. Analytical approximations and diagnostic criteria are derived to predict when standard B-PINNs remain reliable, and experiments confirm both the bias and the effectiveness of the chain formulation.

By Michael Obermayr, Robert Peharz
arXiv AI
Oct 9

Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models

The paper introduces Dynamic Boundary Evaluation (DBE), a method that moves beyond static benchmarks by locating the per‑prompt pass probability near 0.5 for large language models. DBE provides a calibrated item bank with difficulty labels, a Skill‑Guided Boundary Search algorithm that works with API‑level access, and an adaptive evaluation protocol that places new models on a unified ability scale. The approach is applied to safety, capability, and truthfulness tasks, expanding coverage without saturation and remaining compatible with existing datasets.

By Haoxiang Wang, Da Yu, Huishuai Zhang
arXiv Machine Learning
Oct 9

Temporal transformer CAN encoder with federated lightweight heads for anomaly detection

The paper introduces a privacy‑preserving anomaly detection framework for in‑vehicle networks that uses a Temporal Transformer CAN Encoder with Federated Lightweight Heads. It captures subtle temporal and contextual irregularities in CAN bus traffic by learning signal evolution over time, while federated learning allows multiple vehicles or ECUs to collaboratively improve a shared model without sharing raw data. Experiments on open‑source datasets demonstrate robust detection performance with maintained efficiency and privacy.

By Konstantinos Gyftodimos, Kyriakos Chiotis, Elena Politi, George Dimitrakopoulos, Eirini Liotou
arXiv Machine Learning
Oct 9

Explaining the Saliency Map Sparsity of Adversarially-Trained Neural Networks

The paper investigates why saliency maps of adversarially‑trained neural networks appear sparse. It provides a theoretical explanation for two‑layer ReLU networks, showing that as data size and neuron count grow, minimizers converge to a Bayes classifier with minimal gradient and Barron norm. The sparsity arises because α−−‑training with λ−1 attacks yields anisotropic gradient norms that favor axis‑aligned, sparse gradients, a claim supported by experimental evaluation of gradient α‑norms and sparsity thresholds.

By Yannick Lunk, Atell Yehor Krasnopolsky, Damien Garreau, Leon Bungert
arXiv Machine Learning
Oct 9

Gen-PINNs: Generative Adversarial Physics Informed Neural Networks for solving partial differential equations

Gen-PINNs introduces a deterministic residual-adversarial framework that combines Generative Adversarial Networks with Physics-Informed Neural Networks to solve partial differential equations. The method addresses limitations of standard PINNs—such as spectral bias, loss imbalance, and optimizer stagnation—by using a generator that learns the PDE solution with dynamically weighted physics-informed losses and discriminators that assess residuals against zero-residual states. Experiments on nonlinear and higher-order PDEs (Burgers, Allen‑Cahn, Kuramoto‑Sivashinsky) show significant gains in accuracy and convergence, especially for sharp-front, stiff, and high-order problems.

By Muhammad M. Akmal, Kamy Sepehrnoori, Michael J. Pyrcz
arXiv Computation and Language
Oct 9

Measuring Cultural Alignment Beyond the Average: A Framework for Evaluating Maternal-Health LLM Interactions in Indian Contexts

The paper introduces MH‑INDIC, a framework that evaluates maternal‑health language model interactions in North Indian urban and semi‑urban settings by measuring cultural alignment across ten dimensions of reasoning. Using a survey of 102 pregnant and postpartum women, the authors assess ten LLMs, finding that while models approximate overall population‑level cultural patterns, they lack the demographic and household‑specific variation seen in humans. The study also demonstrates that human‑grounded conditioning improves both profile alignment and dialogue quality in generated interactions.

By Umaira Izhar, Gunjan Arora, Pushpendra Singh
arXiv Machine Learning
Oct 9

Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control

The paper introduces Transport REINFORCE, a model‑free policy‑gradient method for discrete‑time mean‑field control that captures both the dynamics and population‑distribution effects of a policy. It perturbs the population distribution—directly on the simplex for finite state spaces and via a Gaussian‑mixture transport map for continuous spaces—to estimate the missing mean‑field contribution. The authors prove consistency of the perturbed objective and gradient, provide bias and mean‑square error bounds, and demonstrate improved performance over standard REINFORCE on several benchmarks.

By Adonis Jamal, Samy Mekkaoui, Yadh Hafsi, Huy\^en Pham