The paper evaluates whether dense, token‑level rewards improve training of language models compared to the standard group‑relative policy optimization (GRPO) algorithm. Using a matched‑compute protocol that equalizes generated tokens and GPU‑hours across interventions, the authors find that dense rewards—such as Monte‑Carlo step‑wise scoring—do not outperform GRPO, and in some cases perform worse even when matched for compute. Instead, a simple strategy of training on prompts predicted to be near the 50% success threshold yields consistent accuracy gains across multiple models and settings.
By Vikash Singh, Weicong Chen, Debargha Ganguly, Yanyan Zhang, Mohsen Hariri, Sreehari Sankar, Wang Yang, Chaoda Song, Alexander Nemecek, Osama Zafar, Yuting Shao, Kai Ye, Biyao Zhang, Xinpeng Li, Shouren Wang, Erman Ayday, Jing Ma, Sanmukh Rao Kuppannagari, Vipin Chaudhary
arXiv:2609.22910v1 Announce Type: new
Abstract: Vision-language agents that crop and zoom are trained with rewards that credit a successful tool call, yet a successful call does not show that the mod...
By Kunyu Peng, Junming Liu, Ruiqi He, Qingzhuo Wang, Jianzhong Qi, Xianhui Liu
arXiv:2607. 18163v1 Announce Type: cross Abstract: PPO and the GRPO baseline studied here use clipped surrogate objectives whose favorable-direction saturation introduces an abrupt change in the scalar objective's derivative.
By Chinmay Rane, Kanishka Tyagi, Michael Manry
arXiv:2607. 16244v1 Announce Type: cross Abstract: Training multi-turn evidence-reading agents with outcome-only reinforcement learning is unstable because intermediate turns receive little direct credit.
By Hao Dou
arXiv:2607. 12640v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards, and Group Relative Policy Optimization (GRPO) in particular, is now run routinely on a supervised checkpoint in the hope of producing a stronger agent.
By Chengguang Gan, Zhixi Cai, Yunhao Liang, Hanjun Wei, Shiwen Ni, Qinghao Zhang
The paper presents an unsupervised multimodal reward model built on a frozen Qwen3‑VL backbone. By adding a linear head to an intermediate layer and training it to rank a web caption higher than a perturbed version of itself, the authors avoid human preference labels and achieve competitive performance—outperforming GPT‑4o on several benchmarks with far fewer parameters and compute. The approach demonstrates that a carefully chosen intermediate layer, simple perturbation-based training, and verifier‑driven fine‑tuning can yield effective vision‑language rewards without costly annotation.
By Yueying Li, Jingxuan Fan, Kiant\'e Brantley, Sham M. Kakade, Hanlin Zhang