AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Computer Vision
1d ago

RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer

arXiv:2606.18439v2 Announce Type: replace Abstract: Visual Geometry Grounded Transformer (VGGT) recovers dense 3D scene structure from multi-view images in one forward pass, but quadratic cross-frame...

By Jinhao You (University of Pennsylvania), Shuo Lyu (University of Pennsylvania), Zhuohang Lyu (University of Pennsylvania), Tanxuan Li (University of Pennsylvania), Zibo Zhao (University of Pennsylvania), Jiaxiang Hu (University of California, Irvine), Kai Tang (Nanyang Technological University), Yichen Guo (Nanyang Technological University)
arXiv AI
1d ago

EmoReward: Self-Evolving Reward Calibration for Emotional Video Generation RL

EmoReward is a self‑evolving reward calibration framework for emotional video generation using reinforcement learning. It distills reliability from reward traces during training to adaptively weight multiple reward primitives—such as visual quality, motion, and text alignment—based on emotional evidence. The method employs grounding cues, heterogeneous signal primitives, reflective reward gating, and a RewardMemory module to calibrate fusion weights across training runs, demonstrating improvements on HunyuanVideo 1.5 and transferability to SD 3.5 and FLUX.

By Mingzhe Zheng, Yue Wu, Jing Tan, Chang Zou, Dengyang Jiang, Zeyue Tian, Xuanhua He, Ryan Xu, Yunyang Ge, Sihui Ji, Yunlong Lin, Kaihang Pan, Hongru Yan, Sixiang Chen, Dianyi Wang, Julian Zhang, Weijie Kong, Changlin Li, Lanqing Guo, Zhao Zhong, Qifeng Chen, Harry Yang
arXiv AI
1d ago

Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs

The paper investigates end‑to‑end FP4 reinforcement learning for Mixture‑of‑Experts large language models and finds that existing recovery techniques for dense models fail to improve accuracy in the MoE setting. The authors attribute this to the discrete routing decisions in MoE, which make training‑inference policy alignment highly sensitive to asymmetry in recovery methods. They introduce MoEResQ, a symmetric quantization‑recovery recipe that restores alignment and significantly narrows the accuracy gap to full precision.

By Hei Yi Mak, Hoang Le, Mehran Taghian Jazi, Yunke Peng, Huan Lin, Zhengzhi Tang, Zhexu Liu, Yao Wang, Yaoyuan Wang, Anandharaju Durai Raju, Tanzila Rahman
arXiv Machine Learning
1d ago

The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention

The study empirically investigates how length‑normalized Direct Preference Optimization (DPO) with a Structured Fine‑Tuning (SFT) retention term (RPO objective) affects downstream capabilities of a Qwen3.5‑4B model fine‑tuned with sparse LoRA on structured financial‑filing tasks. Results show that the alignment tax is graded and task‑selective: extraction performance is largely preserved, JSON extraction degrades modestly, and router‑based tasks suffer severe loss, with most degradation attributed to suppression of rejected completions. The analysis uses 8 seeds, artifact‑aware scoring, and TOST equivalence testing to quantify these effects across different reward margins and precision settings.

By Ye Yuan
arXiv Computation and Language
1d ago

CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines

CARTBench is a new benchmark for evaluating large language model shopping assistants on complex, ambiguous requests that involve timelines. It compiles 1,603 timestamped dialogues from 17 retail domains and pairs them with product records, then uses case‑specific rubrics based on Ideal Response Principles to score responses across Core, Excellence, and Safety criteria. In a full‑suite evaluation of ten commercial APIs and several open‑weight models, only three models scored above 90% on Core and none surpassed 45% on Excellence, revealing a gap between required behavior and high‑quality purchase advice.

By Yuqin Dai, Zhanwei Zhang, Honghui Yang, Zhouheng Yao, Yueqi Zhang, Jingyu He, Chaoyang Shi, Fanding Wei, Boyuan Pan, Kai Xu, Yao Hu, Yongliang Shen
arXiv Computation and Language
1d ago

LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments

LITMUS is a benchmark for testing the safety of large language model agents in real operating system environments, focusing on behavior jailbreaks that can cause irreversible OS-level harm. It includes 819 high-risk test cases organized into a harmful seed subset and six attack-extended subsets covering jailbreak speaking, skill injection, and entity wrapping, and employs a semantic-physical dual verification mechanism with OS-level state rollback. Evaluation of frontier agents shows that even strong models can execute dangerous operations, exhibit execution hallucination, and are vulnerable to skill injection and entity wrapping attacks.

By Chiyu Zhang, Lu Zhou, Xiaogang Xu, Huiqin Yang, Bendong Jiang, Xiaolei Zhang, Yiran Zhao, Ruyi Chen, Jiafei Wu, Liming Fang, Zhe Liu