arXiv AI October 16, 2026 By Licheng Pan, Haocheng Yang, Yuzhou Chen, Yichen Sun, Yuan Lu, Yunfei Teng, Jiajun Fan, Guangyi Liu, Zhixuan Chu, Hao Wang

CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF

Read the original on arXiv AI →

CleanRM is a reward‑modeling approach that mitigates the bias caused by noisy preference data in RLHF. It calibrates sample‑wise noise ratios using a small set of clean feedback via optimal transport, aligning the joint distribution of noisy and clean preferences. The method then trains a reward model with a noise‑corrected surrogate loss, proven to be unbiased with respect to the clean risk, and outperforms existing denoising baselines in experiments.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Jun 17

NoiseTilt: Noise-Tilted Reverse Kernels for Diffusion Reward Alignment

arXiv:2606. 18066v1 Announce Type: new Abstract: We introduce the Noise-Tilted Reverse Kernel (NTRK), a reward-guided diffusion sampler that injects reward gradients through the noise term, leaving the pretrained reverse kernel unchanged and requiring only a single sample per step.

By Jisung Hwang, Yunhong Min, Jaihoon Kim, I-Chao Shen, Minhyuk Sung

diffusion benchmarks safety

arXiv AI

Sep 30

Diffusion Reward Models

arXiv:2609.33803v2 Announce Type: replace-cross Abstract: Reward models underpin the alignment of large language models, yet the dominant designs reduce each prompt--response pair to a point estimate...

By Xiangyang Wang, Bingxiang He, Zeyuan Liu, Jiaze Wang, Ziqing Qiao, Yuxin Zuo, Huan-ang Gao, Cheng Qian, Wenbin Zhang, Ran Li, Youbang Sun, Ning Ding, Yuanchun Shi, Zhiyuan Liu, Chaojun Xiao, Chun Yu

llms diffusion reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Aug 5

Latent Reward Registers for Diffusion Preference Alignment

arXiv:2608. 03929v1 Announce Type: new Abstract: Aligning diffusion models with human preferences usually relies on a sparse terminal reward evaluated on the final generated samples, presenting a severe temporal credit-assignment challenge across the multi-step denoising process.

By Yuanshen Guan, Zipeng Feng, Zhiwei Xiong, Peiqin Sun

llms diffusion reinforcement-learning efficiency benchmarks safety

arXiv AI

Jul 17

Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning

arXiv:2506. 12529v2 Announce Type: replace-cross Abstract: Preference-based Reinforcement Learning (PbRL) entails a variety of approaches for aligning models with human intent to alleviate the burden of reward engineering.

By Sara Rajaram, R. James Cotton, Fabian H. Sinz

reinforcement-learning benchmarks safety

arXiv AI

Jul 9

Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF

arXiv:2607. 07693v1 Announce Type: cross Abstract: Reinforcement learning from human feedback (RLHF) has emerged as a powerful paradigm for aligning generative models with human preferences.

By Eric Zhu, Abhinav Shrivastava, Soumik Mukhopadhyay

diffusion reinforcement-learning

arXiv AI

5d ago

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

The paper introduces Collaborative Reward Modeling (CRM), an online framework that uses cross-model peer review and curriculum learning to identify reliable preference pairs for training reward models (RMs). By having two RMs evaluate each other's sample selections and synchronizing their learning progress, CRM reduces the impact of noisy feedback, improving RM generalization by up to 9.94 points on RewardBench under 40% preference noise. The approach also extends to implicit-reward alignment, demonstrating robustness and broad applicability for learning from imperfect feedback.

By Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Yuming Yang, Chenhao Huang, Tao Gui, Qi Zhang, Xuanjing Huang

llms reinforcement-learning safety

One email a morning, machine-written

Related stories

NoiseTilt: Noise-Tilted Reverse Kernels for Diffusion Reward Alignment

Diffusion Reward Models

Latent Reward Registers for Diffusion Preference Alignment

Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning

Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment