AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv Computation and Language
Sep 30

Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency

Large Language Model judges are commonly used to rank texts via pairwise comparison, with reliability traditionally measured by position bias, transitivity, and pairwise agreement. This paper argues that these proxies are misleading because they are dominated by close‑rank‑gap pairs, which contribute little to the overall ranking, while far‑gap pairs carry the true ranking signal. Experiments on simulations and human‑rated corpora show weak correlation between the proxies and actual ranking accuracy, suggesting judges should be evaluated using rank‑gap‑conditional metrics against human rankings.

By Bruno Brocai, Maria Becker
arXiv Computer Vision
Sep 30

Adversarial Training for Pixel Diffusion

Pixel diffusion models generate RGB images directly but tend to miss fine‑scale natural‑image statistics. The authors introduce an adversarial post‑training step that adds an adversarial loss to the model’s output at non‑high‑noise timesteps, without changing the architecture or sampling procedure. This approach improves distribution fidelity, coverage, prompt alignment, and perceptual quality across two pixel backbones, and restores missing high‑frequency spectral power while avoiding memorization or mode dropping.

By Xin Lin, Zhifei Zhang, Yuqian Zhou, Haitian Zheng, Zhe Lin, Ming-Hsuan Yang, Truong Nguyen
arXiv Computation and Language
Sep 30

Large Language Models Exhibit Human-Like Bayesian Hypocrisy

Large language models (GPT‑4o and Claude 3.7 Sonnet) were tested on two variations of a Bayesian reasoning task across 5 experiments and 5,000 trials. They performed near human levels on the task, but their reasoning was more rule‑based and rigid. The models also exhibited Bayesian hypocrisy, condemning others who used the same Bayesian reasoning, a human‑like error that highlights a dissociation between self‑performance and other‑judgment.

By Nykko Vitali, Mahzarin R. Banaji
arXiv Machine Learning
Sep 30

Introducing the CZAR Loss: A Tailored Objective Function for Financial Log-Return Predictions

arXiv:2609.36061v1 Announce Type: new Abstract: In quantitative finance, standard regression losses are misaligned with the economics of return prediction. As the conditional mean of financial log-re...

By Joel Pfeffer (Allora Foundation), J. M. Diederik Kruijssen (Allora Foundation), Florian Stecker (Allora Foundation), Steven N. Longmore (Allora Foundation, LJMU)
arXiv AI
Sep 30

FLOORA: A Human-Aligned Domain-Specific Language Model for Architectural Design

arXiv:2609.36064v1 Announce Type: cross Abstract: Foundation models are powerful generators, but many engineering domains require structured representations that general-purpose systems handle poorly...

By Sahand Rezaei-Shoshtari, Patryk Wozniczka, Shu Ishida, Gregg Streuber, Farnoosh Javadi, Jeffrey Landes, Angela Ju, Muhammad Azam, Bryan Lim, Johan Luttun, Indrajeet Haldar, Jonathan Shaw, Beatriz Guerra, Ivan Sosnovik, James Stoddart, Robert Giaquinto, Adam Gaier
arXiv Computer Vision
Sep 30

Weeding Out Bad Seeds: Initial-Noise-Robust Unlearning for Text-to-Image Diffusion Models

arXiv:2609.37537v1 Announce Type: new Abstract: Machine unlearning has emerged as a critical post-hoc safety measure to erase sensitive concepts from Text-to-Image (T2I) models without prohibitive re...

By Arian Komaei Koma, Seyed Amir Kasaei, Aida Aryafar, Matin Ghiasi, Ali Aghayari, Amirhossein Souri, Mohammad Mosayyebi, AmirMahdi Sadeghzadeh, Mohammad Hossein Rohban
arXiv Computer Vision
Sep 30

Speed in the Blind Spot: An Interpretability Analysis of Dynamic Perception in VLMs for Autonomous Driving

Vision‑Language Models (VLMs) used in autonomous driving are evaluated on their ability to understand velocity through three tasks: surrounding‑agent speed, current ego speed, and short‑horizon future ego‑speed. Experiments on nuScenes show that while current ego speed can be accessed internally, verbal outputs are poor and surrounding‑agent speed is weakly encoded; temporal cues and frame order are largely ignored. Specialized driving models (Alpamayo‑1.5) improve latent speed representations but still leave gaps between internal knowledge and verbal readout, indicating that good planning outputs do not guarantee reliable dynamic state recovery.

By Katharina Winter, Stefan Englmeier, Fabian B. Flohr
arXiv Machine Learning
Sep 30

GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences

Grounded Transitive RL (GTRL) is an offline goal‑conditioned reinforcement learning algorithm that improves upon divide‑and‑conquer value learning by grounding updates with a one‑step temporal‑difference (TD) target. By adding this TD target rather than replacing it, GTRL ensures every state‑goal pair receives an update and corrects bias from hindsight relabeling through reweighting based on reachability. The method was evaluated on nineteen OGBench tasks across stochastic, deterministic, and stitching environments, achieving the highest average success rate among compared approaches.

By Abdul Monaf Chowdhury, MD Sameer Iqbal Chowdhury, Shifat E Arman, Md Mehedi Hasan
arXiv AI
Sep 30

PowerZooJax: A JAX-based Power System Benchmark for Reinforcement Learning

PowerZooJax is a JAX-based benchmark suite designed for reinforcement learning in power system operation. It offers five constrained Markov decision process tasks covering generation, transmission, distribution, distributed energy resources, and data center microgrids. By implementing power flow, economic dispatch, market clearing, and device dynamics as JAX computation graphs, the entire training and evaluation loop runs on the GPU, yielding significant speedups over CPU-based simulations and enabling standardized evaluation of policy returns, safety violations, and out-of-distribution stress conditions.

By Zhanhua Pan, Xiao Liu, Zhilong Cao, Jianhong Wang, Dawei Qiu
arXiv AI
Sep 30

CoRe: Co-Evolving Reward Models for Mitigating Latent Reward Hacking in Video Diffusion Models

CoRe introduces a co‑evolving reward framework to mitigate latent reward hacking in video diffusion models. By continuously refitting the latent‑reward model on the generator’s current samples and anchoring it to real‑video preferences, CoRe prevents the generator from drifting outside the reward model’s training support. Experiments on Wan2.1‑T2V‑1.3B demonstrate that CoRe improves generation quality over pretrained models and prior alignment methods while avoiding quality collapse.

By Zhaolong Su, Yujin Han, Feng Wang, Jameson Dong, Hins Hu, Difan Zou
arXiv AI
Sep 30

Neural Structural Reasoner: A Brain-inspired Architecture for Reasoning over Structured Knowledge

Neural Structural Reasoner (NSR) is a brain-inspired neural network that preserves relational structure directly in the connectivity and dynamics of coupled neuronal populations. It incorporates multi-layered architecture, stable entity representations, and path integration to perform link prediction by parallelizing over candidate relational structures and using confidence-weighted scores. NSR achieves competitive accuracy on knowledge-graph benchmarks, trains faster than several neural baselines, and offers native interpretability by tracking human-readable neuron activations and extracting latent relational hierarchies and compositional rules.

By Zixing Jia, Yuhang Pan, Ni Ji
arXiv AI
Sep 30

FairDiff: Mitigating the Self-Reinforcing Matthew Effect in Diffusion Recommender Models

FairDiff is a new fairness‑aware diffusion framework designed to mitigate the self‑reinforcing Matthew Effect in Diffusion Recommender Models (DRMs). It introduces Popularity Condition Guidance (PCG) to reweight inference‑time gradients and penalize high‑popularity items, and a Semantic Calibration (SC) module that aligns forward and reverse distributions via optimal transport. Experiments show FairDiff achieves state‑of‑the‑art performance while reducing popularity bias in DRMs.

By Song-Li Wu, Xianquan Wang, Zhaocheng Du, Weinan Gan, Jingyi Wang