AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv AI
Sep 10

PAC-CF: Calibrating Irreversible Frontier Pruning in LLM-Guided Search

PAC-CF introduces a method called Probably Approximately Correct Conformal Filtering to calibrate irreversible frontier pruning in large language model (LLM)-guided search. The approach treats tree pruning as a PAC‑guaranteed decision problem, deriving a conformal margin from score deficits on held‑out Native traces and applying a direct score‑gap filtering rule during deployment. Experiments across multiple domains and controllers show that PAC‑CF improves utility at various budgets and reduces workload metrics, notably boosting equal‑domain utility by 4.38 points and cutting physical requests by 18.95% with a 23.76% token reduction on a pruning‑aware ToolTree under a 100‑request budget.

By Tianhao Qian, Jiayu Chen, Lixu Wang
arXiv Machine Learning
Sep 10

Modal Logic Neural Networks

arXiv:2512.03491v3 Announce Type: replace Abstract: Neural Networks are indispensable to natural sciences and society. Their impact extends from applications in public health to workforce productivit...

By Antonin Sulc, Noor Naddour
arXiv AI
Sep 10

How to Backdoor Image Knowledge Distillation

The paper demonstrates that image knowledge distillation can be backdoored even when the teacher model is clean, by poisoning the distillation dataset with triggered and manipulated images that the teacher already classifies as a target label. The attack, effective at poisoning rates as low as 10%, uses targeted adversarial perturbations and GAN-based class transitions to embed a backdoor into the student model while preserving its performance on clean data. The study highlights that the security of knowledge distillation depends not only on the teacher but also on the integrity of the distillation data.

By Qian Ma, Chen Wu, Prasenjit Mitra, Sencun Zhu
arXiv AI
Sep 10

Suan: Rectifying Direct Preference Safety Alignment in Large Language Models

The paper introduces Suan, a new preference optimization algorithm designed to improve safety alignment in large language models. Suan operates directly at the gradient level, avoiding traditional variational derivations, which yields more interpretable and robust training dynamics. Experiments show that Suan outperforms existing methods, achieving superior safety alignment while maintaining response utility.

By Oleksandr Cherednichenko, Roman Klypa
Hugging Face Trending Papers
Sep 9

Algorithmic stability via ensembling

Algorithmic stability refers to the property of an algorithm being insensitive to perturbations of the input data, where the type of perturbation may vary depending on the setting. In this work, we de...