arXiv AI September 10, 2026 By Oleksandr Cherednichenko, Roman Klypa

Suan: Rectifying Direct Preference Safety Alignment in Large Language Models

The paper introduces Suan, a new preference optimization algorithm designed to improve safety alignment in large language models. Suan operates directly at the gradient level, avoiding traditional variational derivations, which yields more interpretable and robust training dynamics. Experiments show that Suan outperforms existing methods, achieving superior safety alignment while maintaining response utility.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers

Sep 8

Suan: Rectifying Direct Preference Safety Alignment in Large Language Models

Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their und...

llms benchmarks safety

arXiv AI

Jun 2

SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment

arXiv:2606. 02530v1 Announce Type: new Abstract: Aligning Large Language Models (LLMs) with human values often degrades their general capabilities, termed the alignment tax.

By Hao Li, Jingkun An, Zijun Song, Pengyu Zhu, Rui Li, Hao Wang, Wendi Feng, Yesheng Liu, Lijun Li, Jin-Ge Yao, Lei Sha

llms efficiency benchmarks safety

arXiv AI

Aug 14

Cat-DPO: Category-Adaptive Safety Alignment

arXiv:2604. 17299v3 Announce Type: replace-cross Abstract: Aligning large language models with human preferences must balance two competing goals: responding helpfully to legitimate requests and reliably refusing harmful ones.

By Tiankai Yang, Yi Nian, Xinyuan Li, Ruiyao Xu, Henry Peng Zou, Kaize Ding, Xiyang Hu, Yan Liu, Yue Zhao

llms reinforcement-learning safety

arXiv Machine Learning

Jun 4

Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning

arXiv:2603. 07445v2 Announce Type: replace-cross Abstract: Large language models (LLMs) often require fine-tuning (FT) to perform well on downstream tasks, but FT can induce safety-alignment drift even when the training dataset contains only benign data.

By Guoli Wang, Haonan Shi, Tu Ouyang, An Wang

llms fine-tuning safety

arXiv AI

Jun 9

DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment

arXiv:2606. 07678v1 Announce Type: cross Abstract: Safety alignment for large language models relies on preference data, but current pipelines often train on large, redundant datasets.

By Yi Nian, Tiankai Yang, Yudi Zhang, Qi Pan, Zelong Xu, Shenzhe Zhu, Qingqing Luan, Yue Huang, Xiangliang Zhang, Yue Zhao

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 23

Matching Ranks Over Probability Yields Truly Deep Safety Alignment

arXiv:2512. 05518v2 Announce Type: replace-cross Abstract: Open-source Large Language Models (LLMs) play a critical role in the democratization of AI, yet their "open" nature introduces more avenues for malicious actors to misuse them for harmful purposes.

By Jason Vega, Gagandeep Singh

llms fine-tuning safety

One email a morning, machine-written

Related stories

Suan: Rectifying Direct Preference Safety Alignment in Large Language Models

SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment

Cat-DPO: Category-Adaptive Safety Alignment

Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning

DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment

Matching Ranks Over Probability Yields Truly Deep Safety Alignment