AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv AI
Sep 11

Arbitrary Cipher Attacks Against Large Language Models Do Not Require Fine-Tuning

The paper reports that large language models can acquire cipher-based covert communication skills without fine‑tuning, using prompting or in‑context learning instead. This enables new jailbreak attacks that bypass alignment safeguards by encrypting harmful requests, making them appear as nonsensical text to harmfulness classifiers. The authors demonstrate successful attacks against frontier models from Anthropic, Google, and OpenAI.

By Thomas Rivasseau
arXiv AI
Sep 11

Cascading Gradient Inversion via LT-Code Inspired Peeling in Federated Learning

The paper introduces a new gradient inversion attack for federated learning that leverages concepts from erasure‑correcting codes to recover entire training batches and their labels from a single FedSGD round. Unlike previous analytic attacks, this method can exactly reconstruct batches of up to 128 samples on ImageNet and achieves over 90% recovery even when the attacker actively manipulates the model. The study demonstrates that federated learning’s privacy leakage is far greater than previously estimated.

By Saeed Shariati, Mohsen Alambardar Meybodi
arXiv AI
Sep 11

Kernel-Complexity Edge Sanitization for Training-Free Defense against Structural Graph Attacks

Kernel-Complexity Edge Sanitization (KCES) is a training‑free, model‑agnostic defense for Graph Neural Networks that identifies and removes edges with high Kernel‑Complexity (KC) scores, which are indicative of structural influence on the graph kernel complexity metric. KCES leverages a theoretical upper bound on GNN test error derived from the graph Gram matrix to compute edge‑specific KC scores, pruning edges that are empirically enriched with adversarial perturbations. The method is computationally efficient, scalable to large graphs, and consistently outperforms representative robust baselines across diverse attack settings without requiring retraining.

By Yaning Jia, Shenyang Deng, Yaoqing Yang, Chiyu Ma, Wenxuan Xu, Soroush Vosoughi
arXiv AI
Sep 11

BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL

BRACE introduces an anchored Bellman‑residual correction to address stale critic bias in asynchronous reinforcement learning for language models. By limiting the correction horizon to a prefix of policy tokens and adding a constant‑weight Monte‑Carlo tail, it separates policy correction from reward propagation. The method improves mean@1 on BrowseComp‑Plus by 2.4% and runs 2.46× faster per step than synchronous training while staying stable 50 updates off‑policy.

By Guanqun Zhao, Zijun Xie, Binbin Zheng, Jiafeng Lu, Enlei Gong, Zeyu Chen
arXiv AI
Sep 11

Beyond Training: A Feasibility Taxonomy for Inference-Time AI Governance

The paper "Beyond Training: A Feasibility Taxonomy for Inference-Time AI Governance" presents a taxonomy of twenty inference‑time mechanisms for monitoring, verification, and enforcement, each evaluated on a four‑point readiness scale using evidence from four vendors. It applies this taxonomy to a two‑dimensional adversary model and maps the mechanisms to four governance scenarios, finding that most mechanisms are commercially available but only adequate against cooperative or low‑to‑medium‑capability users, not high‑capability state‑level deployers. The study also links inference‑stage controls to hardware‑stage mechanisms through a substitution principle and reports a second‑rater reliability of 0.74. whyItMatters":"The work identifies the current gaps and readiness of inference‑time governance tools, highlighting that existing mechanisms are insufficient against powerful adversaries and thus informing future regulatory and technical development."

By Samar Ansari
arXiv AI
Sep 11

MOONWALK: Mediating Operations with Intent-Evidence-Action Alignment Across Junior-Supervisor Review Workflows in Animation/VFX Pre-Production

MOONWALK is a pre‑production review system for animation and VFX that aligns creative intent, evidence, and action across junior‑supervisor workflows. It records shared intent, anchors judgments to evidence, and translates decisions into clear revision tasks tied to reference notes, with AI handling administrative coordination. A studio study shows that MOONWALK improves intent alignment, decision traceability, and checklist executability compared to a chat‑only interface, while keeping aesthetic authority with practitioners.

By Shih-Yu Lai, Wen-Fan Wang, Sai Ling, Shaune Jan, Bing-Yu Chen, Xiang Anthony Chen
arXiv AI
Sep 11

ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork

The paper introduces ROTATE, a regret-driven open‑ended training framework that jointly improves an Ad Hoc Teamwork (AHT) agent and an adversarial teammate generator. Unlike traditional two‑stage pipelines, ROTATE alternates between enhancing the agent and generating teammates that specifically probe its collaboration weaknesses. Experiments on Overcooked and Level‑Based Foraging show that ROTATE outperforms existing baselines on unseen teammates, setting a new benchmark for robust, generalizable teamwork.

By Caroline Wang, Arrasy Rahman, Benjamin Nativi, Jiaxun Cui, Yoonchang Sung, Peter Stone
arXiv AI
Sep 11

Builder, Defender, Breaker: Measurable Independence and Bounded Autonomy When Generative Models Build, Defend and Test Software

The article discusses how generative models increasingly act as builders, defenders, and breakers of software, challenging the assumption that full autonomy is the ultimate goal. It introduces a framework that defines measurable independence between lifecycle roles based on shared generative substrates, and proposes five autonomy levels, three human roles, and five decision criteria to guide oversight. The authors argue that human authority should focus on specification, accountability, and emergency intervention, and they outline testable hypotheses and protocols to evaluate independence and oversight effectiveness.

By Mohamed Chahine Ghanem
arXiv Machine Learning
Sep 11

When More Is Not Better: Component Anti-Synergy in a P300 Speller

The study investigates whether adding more components to a P300 brain‑computer interface speller always improves performance. Using a full‑factorial experiment on a public dataset, the authors varied Euclidean Alignment, xDAWN spatial filtering, subject calibration, and language model priors, and found that component effects are conditional rather than additive. Calibration was the most influential, while adding components could sometimes reduce performance, demonstrating component anti‑synergy and challenging the assumption that an ‘all‑on’ pipeline is best.

By Lucas Yang, Rui Liu, Fusheng Wang
arXiv Machine Learning
Sep 11

EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compression

The paper introduces EMMI, a framework that enables communication‑efficient inference of multimodal large language models (MLLMs) on edge devices. EMMI encodes each sensor modality separately, fuses the representations, and compresses them into a compact latent vector that is transmitted to a server for high‑capacity reasoning. Experiments on a multimodal benchmark show that EMMI can cut the communication payload by 32× while keeping accuracy comparable, achieving up to a 3.4× reduction in end‑to‑end inference latency under bandwidth‑constrained conditions.

By Motahare Mounesan, Irfan Khan
arXiv AI
Sep 11

PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations

PRAGMA is a benchmark designed to evaluate personalized guidance in long‑term conversations. It includes curated longitudinal conversation histories, evidence annotations, and guidance scenarios that reflect evolving user contexts and incorrect assumptions. Experiments show that current retrieval, memory, and long‑context models struggle to recover relevant conversational evidence and to use it effectively for personalized guidance.

By Hyojeong Yu, Hyukhun Koh, Minsung Kim, Yunah Jang, Kyomin Jung
arXiv Machine Learning
Sep 11

Are We Really Doing Few-Shot Learning? A Critical Examination of Pre-Training Assumptions

The paper critically evaluates common few‑shot learning protocols that rely on pre‑training a model on a large auxiliary set with classes disjoint from the target but drawn from the same visual domain. By comparing no pre‑training, class‑disjoint in‑domain pre‑training, supervised out‑of‑domain pre‑training, and label‑free out‑of‑domain pre‑training across eight datasets and three architectures, the authors find that in‑domain pre‑training yields a 33.41‑point average improvement, while out‑of‑domain pre‑training offers a 23.75‑point gain, revealing a 9.66‑point optimistic bias due to domain overlap. They also demonstrate that a label‑free augmentation strategy can match supervised out‑of‑domain performance and propose a descriptor‑based source‑selection method that closely approximates oracle selection, underscoring the need to move beyond in‑domain pre‑training as the default evaluation protocol.

By Alejandro Galan-Cuenca, Marcelo Saval-Calvo, Antonio Javier Gallego
arXiv Machine Learning
Sep 11

ObstaDiff: Generalizable Diffusion Policy Learning via Obstacle-aware Representations

ObstaDiff is a diffusion-policy framework that introduces a lightweight obstacle-aware visual encoder to generate structured representations of targets, obstacles, and background. By aligning these representations, the policy produces end-effector trajectories that focus on a target-centered bottleneck pose while accounting for surrounding obstacles. In real-robot greenhouse trials, ObstaDiff achieved a 75.41% task success rate and an 8.20% obstacle collision rate, outperforming existing imitation-learning baselines in cluttered agricultural settings.

By Jiawen Wang, Kevin Yao, Khalid Jawed
arXiv Machine Learning
Sep 11

Empirical Evaluation of Membership Inference Attacks on NLP Text Classifiers: A Baseline Study on SST-2

The paper evaluates membership inference attacks (MIAs) on NLP text classifiers using the GLUE SST‑2 sentiment dataset. It compares a TF‑IDF + Logistic Regression pipeline with a fine‑tuned DistilBERT model under a loss‑threshold MIA, finding that both models leak membership signals despite high accuracy. The study also tests mitigations, showing that stronger regularization reduces leakage for Logistic Regression at a utility cost, while fine‑tuning DistilBERT for fewer epochs lowers leakage with minimal accuracy loss.

By William Novak (Minot State University), Muhammad Abusaqer (Minot State University)
arXiv Machine Learning
Sep 11

Rethinking Radiomap Blind Prediction with Limited Environment and Configuration Representations

The paper introduces RadioDecomp, a method for radiomap blind prediction that corrects a prior‑guided predictor using deterministic residual refinement. It identifies the conditional‑mean radiomap as the optimal target under squared loss and decomposes domain risk into approximation error and irreducible uncertainty. Experiments show that the RadioLSR variant excels in cross‑configuration generalization and outperforms a monolithic baseline in cross‑environment scenarios.

By Xiaojie Li, Yu Han, Han Fang, Shangqing Liu, Shi Jin, Chao-Kai Wen
arXiv Machine Learning
Sep 11

Breaking the Central Bias: Spatially Partitioned Experts for Coordinate-Based Neuroevolution

The paper investigates a spatial-concentration bias in Evolvable-Substrate HyperNEAT (ES‑HyperNEAT) when applied to MNIST, where evolved networks focus on a central cluster of input pixels. By partitioning the input image into 13 non‑overlapping spatial segments and evolving a separate expert network for each, the authors achieve a 43% mean accuracy—an 106% relative improvement over the baseline—without relying on data‑driven weighting. The study also introduces a receptive‑field diagnostic to detect silent input‑coverage collapse and a spatial‑partitioning remedy to restore full image coverage.

By Romain Claret, Arthur Gygax, Michael O'Neill, Paul Cotofrei, Michael Palma Mendes, Pascal Felber
arXiv Machine Learning
Sep 11

Differentially Private EEG Feature Anonymization: A Privacy-Utility Case Study in Clinical Neurophysiology

The paper investigates how subject‑level differential privacy (DP) can anonymize EEG‑derived feature representations while preserving clinical utility. It evaluates Gaussian and Laplace perturbations across three deployment scenarios—client‑side, server‑side, and decentralized local training—using statistical utility metrics and a downstream machine‑learning task. Results indicate that DP can be integrated into EEG workflows, but the choice of mechanism, privacy parameters, and sensitivity calibration critically affects data utility, especially in small, imbalanced clinical datasets.

By Noman Sadiq, Mohsen Toorani
arXiv Machine Learning
Sep 11

DNA: Differentially private Neural Augmentation for contact tracing

The paper introduces DNA, a system that enhances decentralized contact tracing by combining statistical inference with a differentially private neural network. By ensuring the neural augmentation satisfies differential privacy, the method improves detection of potentially infected individuals even at a privacy budget of epsilon=1 per message. Simulation results suggest that this approach can reduce infection rates through more targeted testing.

By Rob Romijnders, Christos Louizos, Yuki M. Asano, Max Welling
arXiv Machine Learning
Sep 11

On the Societal Impact of Machine Learning

This PhD thesis examines how machine learning (ML) influences society, noting that ML increasingly shapes consequential decisions and recommendations. It highlights the risk of discriminatory effects when fairness is not explicitly considered in data‑driven systems. The work proposes methods for measuring fairness, decomposing ML systems to anticipate bias, and implementing interventions that reduce discrimination while preserving utility, and it outlines future research directions as ML, including generative AI, becomes more integrated into society.

By Joachim Baumann