AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Machine Learning
Oct 22

Remote Rowhammer Attack using Adversarial Observations on Federated Learning Clients

The paper demonstrates a novel remote Rowhammer attack that targets federated learning (FL) servers by manipulating client sensor observations to induce repetitive clustered memory updates. Using a reinforcement learning attacker, the authors show that they can achieve approximately 70% repeated update rate in FL systems with sparse updates, leading to bit‑flips in the server’s DRAM during tasks such as automatic speech recognition and CIFAR‑10 classification. This work highlights a previously underexplored vulnerability where client‑side adversarial behavior can compromise server memory integrity in FL environments.

By Jinsheng Yuan, Yuhang Hao, Weisi Guo, Yun Wu, Chongyan Gu
arXiv Computation and Language
Oct 22

Output Vector Editing for Memorization Mitigation in Large Language Models

Output Vector Editing (OVE) is a post‑hoc technique that suppresses unwanted text continuations from large language models by editing the output vectors of specific neurons. It uses a two‑stage localization of MLP layers and neurons combined with full‑vocabulary least‑squares updates, and its multi‑anchor variant (OVE‑MA) adaptively selects continuation positions to maximize suppression. Experiments on models ranging from 360 M to 7 B parameters show that OVE‑MA can reduce recoverable continuations by 24.45 % with only five output‑vector updates, outperforming the prior method REVS while requiring far fewer updates.

By Ahmad Dawar Hakimi, Kaiwei Lei, Isabelle Augenstein, Hinrich Sch\"utze
arXiv Machine Learning
Oct 22

CALLIOPE: Private Insights into AI Use, Without Embeddings

CALLIOPE is a new top‑down framework designed to uncover topic hierarchies in large unstructured datasets, such as LLM conversations, while ensuring differential privacy. Unlike conventional methods that depend on clustering vector embeddings, CALLIOPE directly employs LLM capabilities throughout its pipeline. Experiments show that CALLIOPE delivers higher utility than embedding‑based approaches, particularly in high‑privacy scenarios, and its outputs are evaluated using both standard clustering metrics and newly introduced utility metrics.

By Edith Cohen, Vadym Doroshenko, Badih Ghazi, Charlie Harrison, Pritish Kamath, Alexander Knop, Ravi Kumar, Ethan Leeman, Pasin Manurangsi, Mikhail Pravilov, Adam Sealfon, Yurii Sushko, Marika Swanberg, Chiyuan Zhang
arXiv Machine Learning
Oct 22

Unintended Side Effects of Value Training in LLMs

The paper examines how fine‑tuning large language models (LLMs) for specific human values can unintentionally affect other, untrained values and broader behaviors. By adapting four instruction‑tuned LLMs to Hofstede’s value dimensions, the authors evaluate cross‑value transfer and observe side effects on risk preferences, social cognition, and safety traits. They conclude that post‑training value alignment should include comprehensive behavioral assessments and that further research is needed to determine the optimal placement of value training within the development pipeline.

By Siddhesh Pawar, Chen Cecilia Liu, Anna Korhonen, Isabelle Augenstein
arXiv Machine Learning
Oct 22

From Theory to Decision Rule: Calibrating the Noisy-Label Crossover for Vision-Language Model Weak Supervision Across Three Medical-Imaging Benchmarks

The study investigates when adding zero‑shot pseudo‑labels from vision‑language models (VLMs) to small expert‑labelled medical imaging datasets ceases to improve, and may even degrade, classifier performance. Across three benchmarks—PatchCamelyon, ISIC 2019, and NIH ChestX‑ray14—the crossover point occurs after roughly 20–50, 100, and 250–500 expert labels, respectively, with additional pseudo‑labels reducing macro‑AUC by up to 0.14 points. The analysis shows that the VLM’s labeling quality, rather than the student model size, limits performance, and a simple bias‑variance rule can predict the benefit of pseudo‑labels in most settings.

By Bruce Changlong Xu, Jose K. James, Alexander J. Ryu
arXiv Statistics ML
Oct 22

Generalised Robust Bayes for Joint Inference of Model and Contamination

The paper introduces H"older--Bayes, a robust generalised Bayesian framework that jointly infers model parameters and the proportion of contamination without specifying an outlier distribution. By applying the H"older divergence to a scaled model, each posterior draw identifies observations least compatible with the fitted model, and averaging these selections propagates uncertainty to outlier‑detection scores. The authors provide theoretical guarantees—including uniform boundedness of the posterior influence function, an $n^{-1/2}$ bound on posterior‑averaged excess risk, and a Bernstein–von Mises theorem—while also demonstrating practical performance through simulations and real‑data experiments.

By Masahiro Fujisawa, Masaki Adachi, Takuo Matsubara
arXiv Computer Vision
Oct 22

From Canon to Nikon: Camera Domain Shift and Potential Equity Implications in Retinal AI

The study examines how the choice of camera—Canon versus Nikon—affects image quality and AI performance in retinal screening for diabetic retinopathy. In a Brazilian cohort of 8,524 patients, Nikon images were more than twice as likely to be of inadequate quality compared with Canon, despite being cheaper. Using a frozen pretrained retinal foundation model, the authors evaluated 12 disease classification tasks across different resolutions and classifiers, finding that camera-related performance differences vary widely and are not consistently mitigated by higher resolution or classifier choice.

By Lisa Y. W. Tang
arXiv Machine Learning
Oct 22

RL-IF: Tracking Non-local Training Data Influence for Online Reinforcement Learning

The paper introduces RL-IF, a method for tracking non‑local training data influence in online reinforcement learning. It defines a trajectory‑level leave‑one‑out metric (RL‑LOO) and derives a first‑order estimator that accounts for both optimization and policy sampling effects, offering theoretical accuracy guarantees. Using RL‑IF, the authors develop RINSE, a data filtering technique that improves training efficiency and final performance on standard control tasks and an RLHF‑style toxicity‑mitigation scenario.

By Shixuan Liu, Cheng Tang, Yuzheng Hu, Fan Wu, Han Zhao, Jiaqi W. Ma
arXiv Machine Learning
Oct 22

Interpretable Cutting Plane Selection for Mixed-Integer Linear Programming Using Genetic Programming

The paper introduces GPSCS, a learning-based framework that uses genetic programming to evolve a symbolic scoring function for selecting cutting planes in mixed-integer linear programming. It also employs a statistics-based method to decide how many top-ranked cuts to add during each separation round. Experiments on diverse synthetic and real-world benchmarks show that GPSCS reduces solving time by 6% over SCIP’s default strategy and outperforms a neural-network approach and a no-cut baseline, indicating strong generalization and robustness.

By Gwen maudet, Mathis Da Cunha, Gr\'egoire Danoy
arXiv Machine Learning
Oct 22

Circuits of LLM Classifiers

The paper introduces circuits that organize large language model (LLM) classifiers into an adaptive sequence of queries and decisions to improve classification reliability. By treating LLM predictions as noisy measurements of an underlying label, the authors derive the optimal circuit structure, establish fundamental accuracy limits, and offer practical guidance for implementation. Experiments across seven diverse classification tasks—ranging from safety and health to political science and human preference—show that this method outperforms traditional routing and cascading baselines on the cost‑accuracy frontier.

By Hadi Khalaf, Sajani Vithana, Kunal Talreja, Haewon Jeong, Viveck R. Cadambe, Flavio Calmon
arXiv Machine Learning
Oct 22

Is Refusal Direction Universal across Low-Resourced Languages?

The study investigates whether refusal directions—instructions that guide language models to refuse certain requests—are universally transferable across low‑resourced languages. Building on prior work that found near‑perfect transfer among safety‑aligned languages, the authors extend the PolyRefuse framework to Belarusian, Bashkir, and Tajik, and evaluate on Qwen2.5‑14B‑Instruct. Their results show that transfer is graded: directions from safety‑aligned or partially‑aligned languages transfer broadly, whereas those from strongly misaligned languages, such as Yoruba, fail to reliably modulate refusal behavior in other low‑resourced languages once ceiling effects are considered. The study also finds that geometric alignment and behavioral transfer are only partially correlated, complicating the claim of universality and indicating that users of the most misaligned, least‑resourced languages may receive weaker safety guarantees.

By Anastasiia Ivanova
arXiv Machine Learning
Oct 22

SAFTER: Safety-Aware Filtering Through Explicit Recoverability for Robust Reinforcement Learning under Distribution Shift

SAFTER is a safety-aware filtering method that embeds a differentiable backup control barrier function (CBF) into Soft Actor-Critic training, unifying safety, finite-horizon backup safety, terminal recoverability, and input admissibility into a single differentiable certificate. The method achieves an exact minimum-intervention projection without iterative online optimization, leading to higher training success and fewer transitions compared to vanilla SAC and other CBF-filtered baselines. In quadruped navigation experiments, SAFTER outperforms state-of-the-art methods on unseen simulated environments and real-world maps, with ablation studies highlighting the importance of accurate differentiation through the safety filter for robustness.

By Benedictus C. G. Cinun, Bin Hu
arXiv Machine Learning
Oct 22

A Decade of Learning-Based Control: From Theoretic Foundations to Practical Deployment

This tutorial paper reviews a decade of advances at the intersection of machine learning and control, focusing on methods that blend machine learning flexibility with control theory’s structure to create capable, reliable, and safe autonomous systems. It covers core concepts for stability and safety, learning dynamics models to handle uncertainty, and data‑driven policy synthesis techniques such as reinforcement learning, imitation learning, and Bayesian optimization. Representative applications and open challenges are discussed, and the authors provide implementations, examples, and additional resources on their website.

By Angela P. Schoellig, Matthias A. M\"uller, Sebastian Trimpe, Melanie N. Zeilinger, Mahathi Anand, Andrea Carron, Oliver Hausd\"orfer, Victor G. Lopez, Friedrich Solowjow, Haocheng Zhao, Siqi Zhou
arXiv Machine Learning
Oct 22

Certified Neural Approximation of Robust Nonlinear Model Predictive Control

The paper presents a framework for creating neural network controllers that inherit the safety and stability guarantees of a robust model predictive control (MPC) policy. By treating the neural network’s approximation error as a bounded input disturbance and incorporating it into the MPC design, the method ensures recursive feasibility and stability before deployment. The approach is offline and constructive, requiring only Lipschitz bounds, and simulations show the learned controller matches the original MPC’s closed‑loop behavior while reducing online computation.

By Hendrik Alsmeier, Maik Pfefferkorn, Rolf Findeisen
arXiv Machine Learning
Oct 22

Sculpting the Latent Space: Disentanglement with WAE-MMD and Programmable Priors

The paper introduces a new approach to learning disentangled representations by directly sculpting the aggregate posterior using the non‑parametric Maximum Mean Discrepancy (MMD) and programmable priors. It demonstrates that traditional Variational Autoencoders and Adversarial Training consistently fail to enforce a factorized latent space, as quantified by the Latent Predictability Score (LPS). The proposed method achieves state‑of‑the‑art unsupervised statistical independence, a 70% reduction in entanglement, optional semi‑supervised semantic alignment, and precise shaping of the posterior without compromising reconstruction quality.

By Quentin Fruytier, Akshay Malhotra, Shahab Hamidi-Rad, Aditya Sant, Aryan Mokhtari, Sujay Sanghavi
arXiv Machine Learning
Oct 22

Synthesis and Verification of Register Automata over Data Sequences (Technical Report)

The paper presents a method for synthesizing and verifying Deterministic Register Automata (DRAs) over ordered data sequences such as time series and sensor streams. It introduces a local‑search synthesis procedure and an exact robustness checker that runs in polynomial time for a fixed number of registers, reducing robustness to a shortest‑path problem. The framework, implemented in RAExc, is applied to learn interpretable surrogate models for neural sequence classifiers, providing statistical guarantees of fidelity and local stability while verifying robustness on benchmark languages and sequentialized MNIST.

By Chih-Duo Hong, Hongjian Jiang, Anthony W. Lin, Oliver Markgraf, Julian Parsert, Tony Tan
arXiv Machine Learning
Oct 22

Can synthetic data reproduce real-world findings in epidemiology? A replication study using adversarial random forests

The study evaluates whether synthetic data can replicate real epidemiological findings using a new method called adversarial random forests (ARF). Six epidemiological analyses—covering topics such as blood pressure, myocardial infarction, and diabetes—were replicated on ARF-generated synthetic data from three large cohorts, and the results consistently matched the original studies. The authors also show that ARF outperforms other tabular synthesizers in terms of utility, privacy, generalization, and computational efficiency, especially when dimensionality and variable complexity are reduced.

By Jan Kapar, Kathrin G\"unther, Lori Ann Vallis, Klaus Berger, Nadine Binder, Hermann Brenner, Stefanie Castell, Beate Fischer, Volker Harth, Bernd Holleczek, Timm Intemann, Till Ittermann, Andr\'e Karch, Thomas Keil, Lilian Krist, Berit Lange, Michael F. Leitzmann, Katharina Nimptsch, Nadia Obi, Iris Pigeot, Tobias Pischon, Tamara Schikowski, B\"orge Schmidt, Carsten Oliver Schmidt, Anja M. Sedlmair, Justine Tanoey, Harm Wienbergen, Andreas Wienke, Claudia Wigmann, Marvin N. Wright
arXiv Computation and Language
Oct 22

Translation and data ethics

This chapter introduces data ethics for translation, outlining key ethical concerns such as copyright, ownership, and bias that affect all data types. It also highlights specific challenges for minoritized language communities and those on the digital divide, where data can reinforce power imbalances. The authors conclude by discussing responsibility, regulation, and the increasing focus on data ethics within translation studies.

By Joss Moorkens, Monja Jannet
arXiv Computation and Language
Oct 22

Guilt by Association: Semantic Leakage Drives Associative Gender Bias

The paper investigates semantic leakage, where a concept in a prompt influences model output about an entity even when the concept is unrelated, and shows that this effect persists even when the concept is not directly attributed to the entity. Extending this idea to gender bias, the authors demonstrate that gendered concepts can steer language model outputs toward stereotypes simply by being present in the scene, whether or not the subject interacts with them. Their findings suggest that semantic leakage contributes to a form of bias that current evaluation methods may overlook.

By Hazel Chen, Peter West, Hila Gonen