AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Machine Learning
Oct 23

Future of Time Series Anomaly Detection: From Historical Review to Unified Outlook

The paper surveys recent advances in time series anomaly detection (TSAD), highlighting new architectures such as diffusion models, state space models, and large language models. It introduces a refined anomaly taxonomy, evaluates existing metrics through extensive experiments, and releases an open-source benchmark with algorithms, datasets, and evaluation tools. The authors also identify open research challenges, including dynamic thresholding, interpretability, zero‑shot transfer, multi‑modal alignment, and foundational models for time series.

By Zhijie Zhong, Zhiwen Yu, Pengyu Li, Liujian Zhang, Qiaoli Ni, Jiang Jun, Jianming Lv, Kaixiang Yang, C. L. Philip Chen
arXiv Machine Learning
Oct 23

Learning Domain Trajectories with Flow Matching for Gradual Domain Adaptation

The paper introduces FlowGDA, a flow‑matching framework for gradual domain adaptation that models adaptation as continuous transport on a statistical manifold. FlowGDA learns time‑dependent velocity fields to generate smooth domain trajectories, avoiding adversarial Min–Max optimization by using a simulation‑free regression objective. The method incorporates source‑anchored transport, semantic consistency loss, and a two‑stage optimization scheme to prevent semantic drift, achieving superior performance over state‑of‑the‑art techniques.

By Yubo Huang, Yushe Cao, Zixi Wang, Jingzehua Xu, Xin Lai, Xiongyu Hu, Shuai Zhang
arXiv Machine Learning
Oct 23

Post-Hoc Feature Selection Layer for Neural Networks Interpretability

The paper introduces a post‑hoc Feature Selection Layer (FSL) that can be added to frozen pre‑trained neural networks on tabular data to highlight the most important features. This lightweight, trainable module fine‑tunes feature weights based on the original model’s outputs without altering its architecture or parameters. Experiments on synthetic and real‑world datasets show that the post‑hoc FSL preserves predictive performance while improving interpretability, outperforming other post‑hoc methods such as Integrated Gradients and SHAP.

By Jo\~ao Kenji Suwa, Jo\~ao Pedro Silveira e Silva, Bruno Iochins Grisci
arXiv Machine Learning
Oct 23

EgoHand: Monocular 3D Hand Reconstruction from Egocentric Video

EgoHand is a monocular 3D hand reconstruction model that sets new state‑of‑the‑art results on egocentric video. It is trained on a curated 14‑million‑image corpus that blends synthetic data, in‑the‑wild video, and multi‑camera motion capture to balance diversity and supervision quality. The model uses SOMA‑Hand as its parametric representation, processes 2D hand crops with a transformer encoder, and incorporates camera cues and an auxiliary 2D branch to enhance geometric consistency and image alignment, achieving low inference latency and strong out‑of‑domain generalization on benchmarks such as HOT3D, ARCTIC, and H2O.

By Chao-Yeh Chen, Tiffany Chen, Lin Duan, Tianyi Hu, Yu-Hsiang Huang, Umar Iqbal, Jiefeng Li, Edy Lim, Yanchen Liu, Yihan Pang, Hang Qiu, Jun Saito, Shao Su, Qingsi Wang, Xiufeng Xie
arXiv Machine Learning
Oct 23

On the Adversarial Vulnerability of Neural Receivers with Unrolled LDPC Decoders

The paper investigates the adversarial vulnerability of neural receivers that incorporate unrolled LDPC decoders for 5G NR. It finds that standard gradient-based attacks are ineffective due to gradient masking caused by tanh saturation in the BP check-node update, leading to a dramatic decay in gradient norm over iterations. To overcome this, the authors propose two loss functions—L_BP and L_LLR—that restore attack efficacy, significantly increasing block-error rates across different neural receiver implementations.

By Aakash Saini, Jafar Aco Mohammadi, Wolfgang Gerstacker
arXiv Machine Learning
Oct 23

High-Value Training Data from Noisy, Diverse Sources

The paper presents a scalable method for learning dataweights that optimizes training-example importance using metagradients, guided by a small high‑quality target dataset. It introduces parametric weighting rules that generalize to new and streaming data and can transfer from smaller to larger models. Applied to reward modeling, the learned weights improve held‑out performance, adapt models to specialized domains, and enable human‑synthetic supervision mixtures to outperform either source alone.

By Hedi Driss, Evan Frick, Anastasios N. Angelopoulos, Cho-Jui Hsieh, Yejin Huh, Tijana Zrnic, Andrew Ilyas
arXiv Machine Learning
Oct 23

Can Entanglement-Induced Feature-Map Designs Improve Projected Quantum Kernels?

The paper investigates projected quantum kernels (PQKs) and how their performance is often obscured by extensive classical hyperparameter tuning of RBF kernels and SVM regularization. By comparing untuned and tuned PQKs to an RBF baseline, the authors isolate the native inductive bias of PQKs and find that recent designs can be improved by quantum feature maps that use two‑qubit entanglement to capture multivariate correlations in IoT data. The study shows that entanglement‑based topologies provide a stronger native inductive bias, leading to better PQK performance that stems from quantum‑mechanical properties rather than from classical tuning.

By Ravi Kumar Jha, Hsuan-Yin Lin, Eirik Rosnes
arXiv Machine Learning
Oct 23

Optimistic Policy Regularization: Elite-Statistic Reward Shaping

Optimistic Policy Regularization (OPR) is a simple reward‑shaping technique that augments on‑policy agents with a closed‑form bonus derived from elite action priors estimated from top‑quartile episodes. By adding a bounded transform of the log‑ratio of this prior to the policy’s action distribution, OPR requires no extra networks or off‑policy corrections and can be implemented as a one‑line change to PPO’s reward. Experiments on 49 Atari games show that OPR achieves a median human‑normalized score of 148.4 % with only 10 M steps, outperforming published PPO on 36 of 49 games and surpassing self‑imitation learning when matched on frames and backbone.

By Mai Pham, Vikrant Vaze, Peter Chin
arXiv Machine Learning
Oct 23

Graph Memory Transformer (GMT)

The paper introduces the Graph Memory Transformer (GMT), a decoder‑only language model that replaces the traditional Feed‑Forward Network sublayer with an explicit learned memory graph. In GMT, token representations are routed through a bank of centroids linked by a directed transition matrix, using gravitational source routing and token‑conditioned target selection to produce gated displacements instead of retrieved values. The resulting model contains 82.2 M parameters, trains stably, and offers direct inspection of centroid usage and transition structure, though it slightly lags a larger GPT‑style baseline in validation loss and perplexity.

By Nicola Zanarini, Niccol\`o Ferrari, Evelina Lamma
arXiv Machine Learning
Oct 23

SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?

SoundnessBench is a benchmark of 1,099 real research proposals designed to test whether large language models can judge the methodological viability of a research idea before resources are invested. The study evaluates 12 frontier LLMs and finds a pervasive optimism bias, with standard prompting yielding a 74.0% false‑positive rate and aggressive prompting increasing false rejections. Analyses show that this bias persists even when controlling for public corpus contamination, surface features, and literature grounding, indicating that current LLMs are not yet reliable gatekeepers for scientific rigor.

By Sy-Tuyen Ho, Minghui Liu, Huy Nghiem, Furong Huang
arXiv Machine Learning
Oct 23

Risk-Aware Alignment Must Incorporate Reward Uncertainty

The paper addresses policy optimization when rewards are noisy, restricting the policy to stay near a reference policy. It defines risk as the probability of ending up with a worse policy than the reference and shows that, under a Gaussian reward model, the optimal regularizer is a quadratic norm weighted by the reward covariance. Using this insight, the authors propose a new algorithm that regularizes with reward variance estimates, and experiments on synthetic bandits, large language models, and reward models demonstrate that it yields more robust alignment outcomes than existing methods.

By Debangshu Banerjee, Kintan Saha, Aditya Gopalan
arXiv Machine Learning
Oct 23

NADS: Navigator-Guided Data Selection for Mitigating Catastrophic Forgetting in Fine-Tuning

NADS (Navigator‑Guided Data Selection) is a fine‑tuning framework for large language models that mitigates catastrophic forgetting when the original pretraining data are unavailable. It first trains a navigator model without preservation regularization to detect capability drift, then scores synthetic data candidates using the predictive divergence between the navigator and the pretrained model. Finally, it selects a diverse, utility‑aware subset of synthetic data and fine‑tunes while distilling pretrained behavior, achieving a better balance between new‑task performance and general capability preservation with reduced selection overhead.

By Jianhao Zhang, Ou Wu, Yi Du
arXiv Machine Learning
Oct 23

How to Teach Large Multimodal Models New Skills

The paper investigates how to add new skills to large multimodal models (LMMs) without erasing previously learned abilities. By sequentially fine‑tuning on five target skills and tracking performance on eight held‑out benchmarks across three model families, the authors discover that forgetting on one skill can partially recover when the model is later tuned on a different skill. They attribute this to shifts in the output token distribution and propose two simple, selective tuning recipes—updating only self‑attention projection layers or only the MLP Gate&Up while freezing the Down projection—that achieve a better learning‑forgiveness trade‑off than full‑LLM tuning or other common forgetting‑mitigation methods. whyItMatters":"The study shows that carefully choosing which model components to fine‑tune can enable LMMs to acquire new skills while preserving prior knowledge, offering a simpler and more effective alternative to existing forgetting‑mitigation techniques."

By Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem
arXiv Machine Learning
Oct 23

Penalized Fair Regression for Multiple Groups in Chronic Kidney Disease

The paper introduces a penalized fair regression framework that incorporates unfairness penalties for multiple groups, specifically targeting binary outcomes with true positive rate disparity penalties. It can be efficiently implemented via a reduction to cost‑sensitive classification and includes new score functions for selecting penalty weights automatically. Empirical studies show that the method achieves a better fairness‑accuracy trade‑off than existing approaches, and when applied to a national chronic kidney disease dataset it markedly improves fairness across race and ethnicity groups without significantly harming overall model fit.

By Carter H. Nakamoto, Lucia Lushi Chen, Agata Foryciarz, Sherri Rose
arXiv Machine Learning
Oct 23

Giskard : Byzantine Robust and Private Aggregation for Large-Scale Decentralized Learning

Giskard is a scalable protocol for private and Byzantine‑robust decentralized aggregation in large‑scale learning. It organizes participants into a tree of committees of size O(log n) and computes a differentially private coordinate‑wise approximate median using secure multi‑party computation within each committee. Experiments with up to one million participants show that Giskard reduces per‑party communication by nearly three orders of magnitude compared to its closest competitors while maintaining robustness against up to n/4 Byzantine parties.

By Ousmane Touat, C\'esar Sabater, Mohamed Maouche, Sonia Ben Mokhtar
arXiv Computation and Language
Oct 23

SIM-UT: A User Simulation based Evaluation Framework for Medical Triage Chatbots

SIM-UT is a multi‑turn, user‑simulation evaluation framework designed to uncover realistic failure modes in large‑language‑model (LLM) medical triage chatbots. The study applied SIM‑UT to three WhatsApp‑based health chatbots in India, South Africa, and Kenya, comparing a medically fine‑tuned model (MedGemma) with general‑purpose models (ChatGPT and Claude) across more than 5,000 multilingual simulated conversations. A prominent failure identified was de‑escalation risk, where chatbots incorrectly down‑rated serious symptoms after users minimized them, potentially leading to dangerous clinical outcomes.

By Deepali Mittal, Dheeraj Mundhra, Yulia Sidorova, Tersia de Kock, Oli Munns, Sofie S Meyer
arXiv Computation and Language
Oct 23

From Clinical Judgment to Checkable Rules: Clinician-Led Guardrails for Mental-Health Chatbots

The paper presents a clinician‑led safety guardrail for mental‑health chatbots that replaces typical prompt‑engineering or Constitutional AI methods. Two clinicians design, finalize, and adjudicate a rule base of four safety checks, which is formally verified for internal consistency and applied to LLM outputs. The verification layer’s verdicts are recomputable from a named rule and sixteen facts, and it consistently avoids false crisis‑resource demands that a matched LLM judge frequently raises.

By Chenhan Lyu, Kit Wislocki, Sarah Elizabeth Stoeckl, Maitreyi Sinha, Amir Rahmani
arXiv Computation and Language
Oct 23

Group Relative Policy Optimisation Improves Multilingual Speech Recognition in Low-Resource Languages

Group Relative Policy Optimisation (GRPO) is applied to a non‑LLM encoder‑decoder ASR model, Whisper‑large‑v3, to improve multilingual speech recognition across 51 African languages. The method mitigates exposure bias from teacher forcing, especially benefiting low‑resource and noisy audio scenarios. The resulting model, SunflowerASR, sets new macro‑averaged WER records on AfriVox‑v2 (0.227) and SimbaBench (0.372).

By Tim Wenjie Hu, Evelyn Nafula Ouma, Benjamin Akera, John A. Quinn
arXiv Computation and Language
Oct 23

WikiOriginQA: Automated Construction of a QA Dataset with Knowledge Origins for Analyzing Cultural Bias

WikiOriginQA is a newly introduced QA dataset that is automatically constructed by using the creation dates of different Wikipedia language editions to represent the origins of knowledge. The dataset allows for the analysis of cultural bias in large language models, with experiments showing a bias toward Anglophone cultures and highlighting the importance of knowledge propagation into the Anglosphere. This automated approach offers scalable cultural bias evaluation across diverse languages without extensive manual annotation.

By Kento Haneda, Yosuke Kishinami, Ryo Fujii, Keisuke Sakaguchi, Makoto Morishita
arXiv Computation and Language
Oct 23

Invisibilization as a Source of Probability Bias in Language Models

The paper investigates how large language models (LLMs) from OpenAI and Anthropic exhibit probability bias when deciding whether to trust information from reporters of different racial descriptions. Using a two‑urn Bayesian game and 6,600 sessions, the authors find that models often refuse to assess a reporter’s credibility—especially for Black‑described reporters—yet when they do assess, Black reporters receive slightly higher trust. The bias is subtle, growing with reasoning effort, and is not detected by standard audits that compare average trust or only assessed cases.

By Christian Arnold, Ernesto Calvo, Guadalupe Gonzalez, Sebasti\'an Vallejo Vera, Tiago Ventura