The paper introduces MCPO (Margin-Credit Policy Optimization), a method that allocates reinforcement learning rewards to tokens where a language model hesitates, measured by the top‑2 margin of the next‑token distribution. MCPO redistributes each solution’s advantage across its tokens while keeping the mean unchanged, requiring no extra model evaluations. Experiments on MiniCPM5‑2B‑Base and Qwen3‑4B‑Base show MCPO consistently outperforms the baseline GRPO across multiple math benchmarks, especially on harder problems, and encourages longer, higher‑scoring answers.
By Chi Liu, Xin Chen, Yuchuan Weng, Srinivasan Manoharan
arXiv:2610.20497v1 Announce Type: new
Abstract: Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual...
By Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang, Delu Zeng, Jiahao Wang, Junmei Yang, Pan tiewen, Jingfeng Liu, Zhiheng Zhou, Bo Li, Min Chen
arXiv:2610.09804v1 Announce Type: new
Abstract: Reinforcement learning is now central to eliciting reasoning in large language models, while in the popular algorithm Group Relative Policy Optimizatio...
By Yingxiang Yang, Weihang Xiao, Zhunxuan Wang, Joshua Flashner, Niresh Agarwal
The paper critiques the common reinforcement‑learning approach of sampling tool subsets when the full set of tools is enumerable, showing that sampling leads to degraded policy estimates and increased reward sparsity in genomic reasoning tasks. It proposes Full‑Group Policy Optimization (FGPO), which evaluates every tool subset and precomputes rewards in a table, thereby eliminating the need for frozen‑reasoner calls during training. Experiments across five frozen reasoners and three genomic benchmarks demonstrate that FGPO consistently outperforms GRPO, improving average scores by 6.75 points and reducing the number of invoked tools per question.
By Haoyue Liu, Xiaoyu Ma, Ye Chen, Zhichao Wang, Xiaoying Tang
arXiv:2608. 11669v1 Announce Type: cross Abstract: Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer.
By Minglai Yang, Xinyu Guo, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou, Yunzhong He, Daniel Yue Zhang, Ying Liu
arXiv:2606. 18810v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform credit across all tokens, wasting gradient on routine tokens while under-crediting pivotal reasoning steps.
By Yingyu Shan, Yuhang Guo, Zihao Cheng, Zeming Liu, Xiangrong Zhu, Xinyi Wang, Jiashu Yao, Wei Lin, Hongru Wang, Heyan Huang