AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv AI
Oct 7

Palette: A Modular, Controllable, and Efficient Framework for On-demand Authorized Safety Alignment Relaxation in LLMs

arXiv:2605.24154v2 Announce Type: replace Abstract: Current safety alignment of foundation models largely follows a \emph{one-size-fits-all} paradigm, applying the same refusal policy across users an...

By Qitao Tan, Xiaoying Song, Arman Akbari, Arash Akbari, Yanzhi Wang, Xiaoming Zhai, Lingzi Hong, Zhen Xiang, Jin Lu, Geng Yuan
arXiv AI
Oct 7

Intersectional Fairness via Mixed-Integer Optimization

arXiv:2601.19595v2 Announce Type: replace-cross Abstract: The deployment of Artificial Intelligence in high-risk domains, such as finance and healthcare, necessitates models that are both fair and tr...

By Ji\v{r}\'i N\v{e}me\v{c}ek, Mark Kozdoba, Illia Kryvoviaz, Tom\'a\v{s} Pevn\'y, Jakub Mare\v{c}ek
arXiv AI
Oct 7

D3S2: Diffusion-Guided Dataset Distillation for Semantic Segmentation

The paper introduces D3S2, a diffusion‑guided dataset distillation framework tailored for semantic segmentation. It tackles long‑tailed class imbalance, pixel‑wise alignment, and high computational cost by first selecting class‑balanced masks and then synthesizing images with a pretrained diffusion model conditioned on those masks. Guided diffusion sampling further refines the data with segmentation‑consistency and class‑wise feature matching losses, achieving superior performance at a 1% compression rate on ADE20K and COCO‑Stuff.

By Wenjie Zheng, Haoji Hu, Jiali Lu, Xingze Zou, Jing Wang
arXiv AI
Oct 7

Best-of-$N$ Guidance for Test-time Diffusion Alignment

The paper introduces Best‑of‑N Guidance (BoNG), a method that embeds the Best‑of‑N sampling principle directly into the reverse diffusion process of generative models. By performing online BoN selection over denoising particles and using the current best particle as a guidance signal, BoNG steers the particle population toward higher‑reward regions during generation. Empirical results show that BoNG outperforms both standard BoN and SMC across 36 comparisons, achieving the best performance in 29 cases and supporting multi‑output generation with significant speedup.

By Richard Lee Kim, Yeongmin Kim, Gyuwon Sim, Taekyu Kim, Minsang Park, Il-chul Moon
arXiv Machine Learning
Oct 7

Sample-Optimal Estimation of the Fr\'echet Inception Distance

The paper investigates the sample complexity of estimating the Fréchet Inception Distance (FID) between high‑dimensional Gaussian distributions when one distribution is known. It establishes tight bias and variance bounds for the empirical plug‑in estimator, shows how higher‑order extrapolation reduces bias, and introduces a new Relative Taylor Debiasing (RTD) algorithm that achieves optimal sample complexity. Experiments on synthetic data and ImageNet confirm the theoretical predictions and demonstrate RTD’s superior accuracy with limited samples.

By Ziyun Chen, Jerry Li, Kevin Tian, Yusong Zhu