The paper introduces Proximal Entropy Policy Optimization (PEPO), a method that assigns token‑level advantages in reinforcement learning for language models based on a local measure of entropy called proximal entropy. Unlike prior approaches that use global token entropy, PEPO evaluates importance relative to neighboring tokens, making it invariant to prompt difficulty and positional effects. Experiments on Qwen3 and Llama‑3.2 models show PEPO outperforms GRPO and other entropy‑based baselines in mathematical reasoning tasks.
By Yun Kim, Nojun Kwak
arXiv:2606. 04560v1 Announce Type: cross Abstract: Reinforcement learning from verifiable rewards with GRPO is a standard approach for post-training reasoning LLMs.
By Gyeongtae Yoo, Sanghyeok Park, Soohyuk Jang, Ik-hwan Kim, Sungroh Yoon
The paper introduces Grounded Credit Assignment (GCA) for reinforcement learning in multi‑turn search agents. GCA separates credit for evidence retrieval and answer correctness, correcting the bias in GRPO that favors evidence‑retrieving rollouts only proportionally to accuracy gains. Applying GCA to Search‑R1 boosts average EM by 5.7–6.3 points and increases the proportion of evidence‑supported correct answers from 76.9% to 93.4%.
By Qirui Liu, Hao Chen, Yi Wen, Weijie Shi, Yao Zhao, Xianhe Chen, Hao Zhao, Xing Tang, Weihong Luo, Yuyu Luo
arXiv:2610.20497v1 Announce Type: new
Abstract: Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual...
By Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang, Delu Zeng, Jiahao Wang, Junmei Yang, Pan tiewen, Jingfeng Liu, Zhiheng Zhou, Bo Li, Min Chen
arXiv:2609.06107v1 Announce Type: new
Abstract: Data policies for reinforcement learning with verifiable rewards (RLVR) determine which rollouts are used, how strongly they are weighted, and which do...
By Hao Liang, Mingrui Chen, Hengyi Feng, Meiyi Qiang, Wentao Zhang
arXiv:2606. 16154v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but GRPO-style optimization remains prone to collapse.
By Prasanth YSS, Zhichen Ren, Rasa Hosseinzadeh, Ilan Gofman, Yuqi Chen, Zhaoyan Liu, Guangwei Yu, Jesse C. Cresswell, Satya Krishna Gorti
The paper introduces UECR-GRPO, a method that unifies on‑policy distillation and verifier‑based reinforcement learning for mathematical reasoning. It combines verifier rewards and teacher‑derived log‑ratios into a single KL‑regularized objective (Path‑Utility Unification) and then redistributes credit at the token level using entropy‑calibrated redistribution, preserving total task credit. Experiments on five benchmarks show that UECR‑GRPO improves average accuracy by up to 0.89 percentage points over the best baseline for both Qwen3‑1.7B and Qwen3‑4B students.
By Jie Zhang, Jingxiao Yang, Zhehao Huang, Yuhang Liu, Xiaolin Huang
HISPO introduces a segment‑level policy‑optimization technique for reinforcement learning with verifiable rewards, creating entropy‑derived contiguous segments during rollout and applying clipped importance‑sampling correction at this granularity. It bridges the gap between token‑level and sequence‑level corrections, offering a middle‑ground approach for credit assignment in long‑form mathematical reasoning. Evaluated on Qwen3‑1.7B‑Base across six benchmarks, HISPO consistently outperforms or matches the strongest baselines in Pass@8 and Acc@8 metrics, notably improving AIME25 scores over GRPO and GSPO.
By Quoc-Vinh Lai-Dang, Hyo-Sang Shin
The paper introduces Selective Successor Eligibility Traces (S-trace), a critic‑free eligibility trace method designed to improve temporal credit assignment in Reinforcement Learning with Verifiable Rewards (RLVR). S-trace reweights successor importance to recover downstream credit signals under PPO-style clipping and selectively propagates them to historical tokens using entropy-based token selection. Experiments on Qwen3 models show that S-trace yields significant absolute gains in average pass@16 over GRPO and trace-style GRPO, while also producing shorter responses.
By Chaoli Mou, Xinning Chen, Zhan Zhuang, Qinghe Zheng, Deyuan Kong, Yu Zhang
arXiv:2601. 03895v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) has emerged as a popular algorithm for reinforcement learning with large language models (LLMs).
By Chi Liu, Xin Chen
arXiv:2610.34951v1 Announce Type: cross
Abstract: Recent methods for Reinforcement Learning with Verifiable Rewards (RLVR) use rollout diversity to improve reasoning performance or training efficienc...
By Sohyun Son, Yerim Hong, Jaeyun Park, Mujeen Sung
arXiv:2609.38850v1 Announce Type: new
Abstract: Decision models such as Jev answer questions with probabilities, which are only useful if they are calibrated. Open-source reproductions rely on superv...
By Zhimin Gao, Pichao Wang