← Back to all news
arXiv Machine Learning By Fei Ding

Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery

Read the original on arXiv Machine Learning →

The paper investigates how advantage scale calibration can become imbalanced in group-relative optimization when rewards have low variance. It distinguishes two problematic low‑variance scenarios—sub‑resolution jitter and small but credible cardinal gaps—and introduces a three‑way calibration interface that links within‑group scale to reward‑branch strength, prompt‑level batch weight, and KL calibration. Building on this interface, the authors propose the Reward‑Resolution Protocol and MaxNorm‑AC to filter out sub‑resolution gaps and provide bounded recovery for credible gaps, achieving better performance than existing robust‑scale baselines across dense, MoE, math, and code reasoning tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Jun 5

SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Matter

arXiv:2606. 05800v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) often adopts GRPO-style group-relative updates, sampling multiple rollouts per prompt to construct normalized learning signals.

By Powei Chang, Jinpeng Zhang, Chaoqun Sun, MiniWell Tsao, Lianrui Li, Jianxiang Xiang, Chenyu Wang, Yukang Gao, Dongying Kong
reinforcement-learningbenchmarks
More like this →
arXiv Machine Learning
Jun 8

Leveraging Error Diversity in Group Rollouts for Reinforcement Learning

arXiv:2605. 17333v2 Announce Type: replace Abstract: Reinforcement Learning from Verifiable Rewards (RLVR) typically samples multiple responses per prompt and assigns binary rewards based on individual correctness, yet the collective structure of the group output, specifically the distribution of errors, is largely discarded.

By Wenpu Liu, Yuqi Xu, Weichu Xie, Yongfu Zhu, Shuai Dong, Ziyue Wang, Wenqi Shao, Xiaoying Zhang, Tong Yang, Nan Duan, Jiaqi Wang
reinforcement-learningbenchmarks
More like this →
arXiv Machine Learning
Sep 2

Group Adaptive Clipping Policy Optimization

Group Adaptive Clipping Policy Optimization (GAPO) is a plug‑in modification to GRPO methods that adapts the importance‑sampling clipping boundary based on rollout advantage. By allowing rollouts with larger learning signals to receive proportionally greater update headroom, GAPO addresses the limitation of fixed clipping that suppresses rare but informative rollouts. Experiments on Qwen and Llama models show that GAPO consistently improves Pass@1 and Pass@k on math reasoning and coding benchmarks where base model pass rates are low.

By Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan, Rein Houthooft
llmsreinforcement-learningbenchmarks
More like this →
arXiv Machine Learning
Jun 26

Reinforcement Learning without Ground-Truth Solutions can Improve LLMs

arXiv:2606. 27369v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) for training LLMs typically rely on ground-truth answers to assign rewards, limiting their applicability to tasks where the ground-truth solution is unknown.

By Yingyu Lin, Qiyue Gao, Nikki Lijing Kuang, Xunpeng Huang, Kun Zhou, Tongtong Liang, Zhewei Yao, Yi-An Ma, Yuxiong He
llmsreinforcement-learningbenchmarks
More like this →