arXiv AI By Vikash Singh, Weicong Chen, Debargha Ganguly, Yanyan Zhang, Mohsen Hariri, Sreehari Sankar, Wang Yang, Chaoda Song, Alexander Nemecek, Osama Zafar, Yuting Shao, Kai Ye, Biyao Zhang, Xinpeng Li, Shouren Wang, Erman Ayday, Jing Ma, Sanmukh Rao Kuppannagari, Vipin Chaudhary

Are Dense Rewards Worth Their Tokens in GRPO?

Read the original on arXiv AI →

The paper evaluates whether dense, token‑level rewards improve training of language models compared to the standard group‑relative policy optimization (GRPO) algorithm. Using a matched‑compute protocol that equalizes generated tokens and GPU‑hours across interventions, the authors find that dense rewards—such as Monte‑Carlo step‑wise scoring—do not outperform GRPO, and in some cases perform worse even when matched for compute. Instead, a simple strategy of training on prompts predicted to be near the 50% success threshold yields consistent accuracy gains across multiple models and settings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
6d ago

Margin-Guided Token Credit for Reinforcement Learning with Verifiable Rewards

The paper introduces MCPO (Margin-Credit Policy Optimization), a method that allocates reinforcement learning rewards to tokens where a language model hesitates, measured by the top‑2 margin of the next‑token distribution. MCPO redistributes each solution’s advantage across its tokens while keeping the mean unchanged, requiring no extra model evaluations. Experiments on MiniCPM5‑2B‑Base and Qwen3‑4B‑Base show MCPO consistently outperforms the baseline GRPO across multiple math benchmarks, especially on harder problems, and encourages longer, higher‑scoring answers.

By Chi Liu, Xin Chen, Yuchuan Weng, Srinivasan Manoharan
arXiv AI
Sep 11

Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection

The paper critiques the common reinforcement‑learning approach of sampling tool subsets when the full set of tools is enumerable, showing that sampling leads to degraded policy estimates and increased reward sparsity in genomic reasoning tasks. It proposes Full‑Group Policy Optimization (FGPO), which evaluates every tool subset and precomputes rewards in a table, thereby eliminating the need for frozen‑reasoner calls during training. Experiments across five frozen reasoners and three genomic benchmarks demonstrate that FGPO consistently outperforms GRPO, improving average scores by 6.75 points and reducing the number of invoked tools per question.

By Haoyue Liu, Xiaoyu Ma, Ye Chen, Zhichao Wang, Xiaoying Tang
arXiv AI
Aug 13

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

arXiv:2608. 11669v1 Announce Type: cross Abstract: Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer.

By Minglai Yang, Xinyu Guo, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou, Yunzhong He, Daniel Yue Zhang, Ying Liu
arXiv AI
Jun 18

Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards

arXiv:2606. 18810v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform credit across all tokens, wasting gradient on routine tokens while under-crediting pivotal reasoning steps.

By Yingyu Shan, Yuhang Guo, Zihao Cheng, Zeming Liu, Xiangrong Zhu, Xinyi Wang, Jiashu Yao, Wei Lin, Hongru Wang, Heyan Huang