AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,896 stories · RSS feed

arXiv Machine Learning
Aug 20

SCORE: Subject Coordinate Recovery for Label-Free Cross-Subject EEG-to-Image Retrieval

SCORE: Subject Coordinate Recovery for Label-Free Cross-Subject EEG-to-Image Retrieval proposes a new framework that aligns EEG signals from different subjects into a common image space without requiring labeled calibration data. By training on source subjects and estimating an orthogonal transformation at deployment, SCORE recovers target EEG coordinates and selects reliable EEG-image landmarks through hubness-corrected matching. The method achieves state‑of‑the‑art Top‑1/Top‑5 accuracy on two public benchmarks, outperforming existing baselines by significant margins.

By Zhenyao Cui, Siyuan Kan, Siyang Li, Ziwei Wang, Dongrui Wu
arXiv AI
Aug 20

When Clean Signals Are Not Enough: Detecting Structural Ambiguity for Safe Wearable Stress Classification

The paper introduces the Individual Conformal Coupling Monitor (ICCM), a lightweight pre‑inference tool that detects structural ambiguity—when physiological signals that appear plausible individually form a pattern poorly supported by a person’s non‑stress baseline—in wearable stress classifiers. On the WESAD dataset, a Random Forest achieves high mean accuracy but fails entirely for Subject 14 due to weakened cross‑signal coupling near stress onset. ICCM quantifies subject‑specific coupling divergence and can route data to classify, defer, or abstain, reducing false positives slightly and withholding some misclassified windows, though it does not fully correct the failure.

By Saba A. Farahani, Hung Cao, Amir M. Rahmani
arXiv Machine Learning
Aug 20

Safe Domain Adaptation for Physics: Overcoming Nuisances, Label Shifts, and Simulation Priors

The paper introduces a new approach to domain adaptation in physics, addressing the fact that simulations often differ from experimental data not only in nuisances but also in the target quantity distribution. By studying a toy air‑shower benchmark with separate nuisance, simulation, and spectrum shifts, the authors show that standard adversarial adaptation can misalign spectra, leading to bias. They propose adaptive domain adaptation that reweights simulated events to focus on genuine physical mismatches and provide a label‑free rule for selecting the best model configuration.

By Ivan Kharuk (Institute for Nuclear Research of the Russian Academy of Sciences, Moscow Institute of Physics and Technology)
arXiv Machine Learning
Aug 20

Multi-Agent Off-Policy Deep Reinforcement Learning for Smart Campus Coverage

The paper studies optimal placement of millimeter-wave base stations in a realistic, non-convex campus layout using deep reinforcement learning. It compares four DRL methods—single-agent DQN, multi-agent partitioned DQN, single-agent DDPG, and multi-agent partitioned DDPG—and finds that the multi-agent DDPG approach achieves full coverage, a Jain's fairness index of 0.94, and superior performance in dense scenarios with 400 users. The multi-agent DDPG also converges more efficiently than single-agent methods.

By Omar Rady, Mohamed Ayman, Ali Arafa, Mohamed Shalma
arXiv Machine Learning
Aug 20

You Are What You Prompt: Prompt Quality, Domain Shift, and Uncertainty in Agrifood Vision-Language Models

The paper studies how prompt quality affects vision‑language models in the agrifood domain. It evaluates Zero‑Shot Prompt Ensembling (ZPE) on CLIP and SigLIP across four datasets, showing that ZPE offers limited gains on in‑distribution data but significantly improves accuracy and calibration when the domain shifts. The authors also introduce Prompt‑based Inconsistency Detection (PID), which uses prompt disagreement to detect failures under severe domain shift, outperforming standard confidence measures.

By Andrea Morales-Garz\'on, Salvador L\'opez-Joya, Miguel L\'opez-P\'erez, Maria J. Martin-Bautista
arXiv Machine Learning
Aug 20

Flux-form spatiotemporal neural operators for coarse-grained dynamics of multiscale PDEs

The paper introduces flux‑form spatiotemporal neural operators for predicting coarse‑grained dynamics of multiscale PDEs without relying on closure models. It learns a surrogate evolution operator from filtered high‑fidelity data, using Fourier convolution for spatial mixing and a causal kernel with time‑lag attention for temporal mixing. The method incorporates a flux‑form inductive bias to maintain conservation and provides a data‑driven rule for selecting memory length, achieving stable, accurate long‑horizon rollouts on benchmark equations and turbulent flow simulations.

By Junfeng Chen
arXiv Machine Learning
Aug 20

Geometric Data Perturbation with Noisy-Anchor Alignment for Privacy-Preserving Collaborative Learning

Geometric Data Perturbation (GDP) allows participants to share distance‑preserving transformations of their private data for one‑shot collaborative learning. The paper examines the vulnerability when an analyst colludes with participants, showing that shared‑anchor alignment can restore compatibility but also enables exact data recovery. To mitigate this, the authors propose adding noise to the anchor representations rather than the private data, demonstrating through experiments on MNIST and CelebA that this approach yields better privacy‑utility trade‑offs under collusion.

By Keiyu Nosaka, Yamato Suetake, Yuichi Takano, Yukihiko Okada, Akiko Yoshise
arXiv AI
Aug 20

Robust Risk Under Evolving Uncertainty: A Wasserstein Counterpart of the Entropic Value-at-Risk

The paper introduces the Wasserstein entropic value-at-risk, a coherent risk measure that replaces the relative-entropy ball of the traditional entropic value-at-risk with an optimal-transport ball. This new measure captures reachable catastrophes that the original entropic measure ignores, and its variational dual mirrors the entropic formula with a transport price replacing inverse temperature. By driving the transport radius with belief entropy, the authors derive a closed‑form robust dynamic‑programming operator whose cautiousness decreases as belief sharpens, providing a certified safety sandwich and a sharp safety switch.

By Deep Kumar Ganguly, Jan K\v{r}et\'insk\'y
arXiv Machine Learning
Aug 20

Beyond Trial Averaging: Anchoring Neural and Visual Representations for Few-Repetition Brain-to-Image Retrieval

The paper introduces NEAR, a neural-anchor-based retrieval framework that improves brain-to-image retrieval when only a few neural trials are available. By treating a high‑repetition center as an anchor, NEAR uses a denoiser to pull noisy queries toward the anchor and a small network to predict pseudo anchors for candidate images, thereby aligning both neural and visual representations. Experiments on EEG, MEG, and fMRI datasets show consistent gains, including a 5.7–9.3 percentage point improvement in 200‑way Top‑1 accuracy on THINGS‑EEG2 with only one or four repetitions.

By Zhenyao Cui, Siyuan Kan, Dingkun Liu, Dongrui Wu
arXiv AI
Aug 20

Different Facets of Verbalised Overconfidence: an Interpretability Study

The study examines how the large language model Qwen3-4B exhibits overconfidence across three expressions of uncertainty—verbal epistemic markers, abstention, and numeric confidence scores—using controlled reasoning scenarios that vary logical necessity and possibility. Findings confirm a pronounced overconfidence tendency, especially when numeric scores are requested, and reveal that certainty is generated by a broad coalition of shared features while uncertainty relies on a sparse set of dedicated features. By intervening on these uncertainty features, the researchers demonstrate causal evidence of the imbalance driving overconfidence and show that this intervention can mitigate overconfident errors across different settings, languages, and an out‑of‑distribution modality task.

By Davide Mazzaccara, Leonardo Bertolazzi, Raffaella Bernardi
arXiv AI
Aug 20

Coupled-cluster molecular properties across the main group that extrapolate beyond training size

The paper introduces MEHnet-MG, an equivariant neural network that predicts a one‑electron Hamiltonian from a single inexpensive B3LYP/def2‑SVP calculation and uses it to compute a wide range of molecular properties—energy, optical gap, dipole, quadrupole, polarizability, Mulliken charges, and Mayer bond orders—at coupled‑cluster accuracy for nine main‑group elements, including phosphorus, sulfur, and chlorine. Trained on a new CCSD(T) dataset, the model reduces property errors by factors ranging from 3.8 to 230 compared to various DFT methods while adding only ~25 ms per molecule. Importantly, by deriving properties from a predicted Hamiltonian rather than pooling atomic features, the architecture naturally incorporates correct size‑scaling, enabling accurate extrapolation to large π‑conjugated systems (up to 58 atoms) where traditional pooling‑based models fail.

By Wenhao He, Xu Chen, Noah Song, Haowei Xu, Tim S. Hindges, Bohan Li, Zihan Lin, Yu Yao, Avetik R. Harutyunyan, Fang Liu, Yao Wang, Hao Tang, Ju Li
arXiv AI
Aug 20

Self- and Other-Labels Induce Bidirectional Bias in LLM Judges

The study investigates bias in large language model (LLM) judges by having ten LLMs evaluate narrative constraint selections rather than generated text. Results show that self-preference largely disappears under blind evaluation when quality and evaluator severity are controlled, but self- and other-labels alone shift scores bidirectionally when quality is matched. The authors conclude that authorship attribution drives evaluation bias and that open-ended, ground‑truth‑free tasks can effectively study LLM judge behavior.

By Songeun Chae, Min Kim, Donghoon Jung, Seojin Choi, Seohyon Jung
arXiv AI
Aug 20

`From Prompt to Perturbation': An Adaptive Framework for Voice-Based Jailbreaks on Audio LLMs

The paper introduces an adaptive jailbreak attack framework that evaluates both cascaded pipelines and end‑to‑end large audio‑language models (LALMs) under a unified setting. It employs a feedback‑guided mutation engine to automatically generate and refine jailbreak candidates across textual prompts and audio perturbations, thereby broadening attack diversity. Experiments on six audio‑based systems show that both paradigms remain highly vulnerable, with the framework achieving higher attack success rates than existing methods.

By Linghan Huang, Bo Li, Huaming Chen, Kim-Kwang Raymond Choo
arXiv AI
Aug 20

Large Language Models in Mental Health: A Systematic Review of Applications, Innovations, and Ethical Challenges

The paper reviews how large language models are applied in mental health, covering areas such as social media analysis, clinical conversational agents, therapy support tools, prompt engineering, and multimodal learning. It synthesizes interdisciplinary studies that use social media posts, electronic medical records, and multimodal inputs to detect depression, assess suicide risk, provide personalized therapy, and generate psychoeducational content. The review also discusses advances in model interpretability, annotation strategies, multimodal fusion techniques, and highlights ethical, sociotechnical, and regulatory challenges while proposing frameworks for safe, equitable, and accountable deployment.

By Yisong Chen, Yifan Gao, Sijing Yu, Chuqing Zhao, Yang Lu
arXiv Machine Learning
Aug 20

To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization

The paper introduces CurriPO, a tree‑structured curriculum that automatically adapts to diverse user reward models in AI alignment tasks. By exploiting the natural hierarchy between easy‑ and hard‑to‑optimize reward models, CurriPO covers a broad user population in a single traversal, reusing previously incorporated reward models. Experiments on personalized continuous control show that CurriPO improves population satisfaction by 1.2–2.1× over the strongest baseline while cutting training time and better serving users traditionally underserved by conventional optimization.

By Taehyung Kim, Jongeun Choi
arXiv Machine Learning
Aug 20

Decision-Metric Alignment in Latent World Models: Diagnostics and Action-Conditioned Objectives for MPC Planning

The paper introduces the concept of decision‑metric alignment, which ensures that Euclidean distance to a goal latent in JEPA‑style latent world models correctly ranks action sequences for model‑predictive control. It proposes two metrics—Plan‑Real Spearman and CEM‑stage Spearman—to evaluate latent–real rank agreement, and identifies encoder distortion, terminal rollout error, and candidate margins as key factors affecting alignment. Building on these insights, the authors present DA‑LeWM, an enhanced latent world model that incorporates inverse‑dynamics and demonstration‑conditioned goal‑action heads, leading to faster convergence and higher online success rates compared to the baseline LeWM while maintaining similar probe scores.

By Jiawei Wang, Ke Rui, Yushen Zuo, Yichun Feng, Minglei Li
arXiv AI
Aug 20

Denoising-Aware Inversion: Revealing Privacy Risks in Noise-Protected Text Embeddings

The paper investigates the vulnerability of Gaussian‑noised text embeddings to inversion attacks. It identifies a "Double Noise Trap" that hampers standard generative methods and introduces DAEI, a denoising‑aware pipeline that significantly outperforms existing baselines in reconstructing original text. Experiments show DAEI improves BLEU by 154% and token‑level metrics by 32–60%.

By Yubo Wang, Shujie Cui, James Bailey, Hongzhi Yin, Wenyu Liang, Min Tang, Shiyue Qin, Weiqing Wang
arXiv AI
Aug 20

Decomposing Wrong-Consensus Agreement in LLM Self-Consistency: A GPT-4.1 Case Study

The paper introduces a pluralistic agreement index, Gamma, to quantify how often wrong runs of large language models (LLMs) agree with the majority consensus. By decomposing Gamma into a mechanical component and a preference‑unexplained residual, the authors show that on GPT‑4.1 the mechanical part explains most of the agreement on multiple‑choice benchmarks but only about half on open‑domain tasks, revealing a residual bias that can cause self‑consistency to backfire on hard questions. The study provides a quantitative framework for understanding when majority voting over LLM samples improves or harms accuracy, without proposing new voting methods.

By Lizhuo Zhang, Mengmeng Tang, Chenfeng Long, Xiaoyong Tang, Xiang Luo
arXiv AI
Aug 20

Do Large Language Models Hallucinate Electric Fata Morganas?

The paper investigates why large language models (LLMs) produce hallucinations—outputs that are fabricated, unverifiable, or contradictory to source material—and argues that these hallucinations have philosophical implications for machine consciousness. It reviews known causes such as source‑target divergence, training‑inference discrepancies, and overfitting, and presents two empirical studies: one showing that higher temperature settings in GPT models yield plausible but incorrect answers, while lower temperatures produce accurate ones; and another demonstrating that an encoder‑only model trained on encyclopedic data answers factually without embellishment, suggesting hallucinations arise from exposure to subjective, socially diverse data rather than cognitive ability. Drawing on Turing, Searle’s Chinese Room, the frame problem, and cybernetic theory, the authors contend that a model’s self‑reports of emotion or sentience fall within the definition of hallucination, implying that any future machine consciousness may remain epistemically inaccessible because it would be indistinguishable from an advanced hallucination.

By Kristina \v{S}ekrst
arXiv AI
Aug 20

MR-IQA-2: Faithful Image Quality Reflection via Fine-Grained Credit Assignment

MR‑IQA‑2 introduces an actor‑editor‑judge framework that separates reasoning from rating in image quality assessment. The actor generates quality reasoning, the editor modifies the image based on identified factors, and a frozen judge compares the original and edited images to provide reflective supervision. Fine‑grained credit assignment allows distinct supervision signals for reasoning and rating, achieving competitive human‑aligned ratings while offering richer, faithful visual understanding.

By Yuan li, Youyuan Lin, Chenhui Chu, Shin'ya Nishida