HISPO introduces a segment‑level policy‑optimization technique for reinforcement learning with verifiable rewards, creating entropy‑derived contiguous segments during rollout and applying clipped importance‑sampling correction at this granularity. It bridges the gap between token‑level and sequence‑level corrections, offering a middle‑ground approach for credit assignment in long‑form mathematical reasoning. Evaluated on Qwen3‑1.7B‑Base across six benchmarks, HISPO consistently outperforms or matches the strongest baselines in Pass@8 and Acc@8 metrics, notably improving AIME25 scores over GRPO and GSPO.
By Quoc-Vinh Lai-Dang, Hyo-Sang Shin
The paper introduces Selective Successor Eligibility Traces (S-trace), a critic‑free eligibility trace method designed to improve temporal credit assignment in Reinforcement Learning with Verifiable Rewards (RLVR). S-trace reweights successor importance to recover downstream credit signals under PPO-style clipping and selectively propagates them to historical tokens using entropy-based token selection. Experiments on Qwen3 models show that S-trace yields significant absolute gains in average pass@16 over GRPO and trace-style GRPO, while also producing shorter responses.
By Chaoli Mou, Xinning Chen, Zhan Zhuang, Qinghe Zheng, Deyuan Kong, Yu Zhang
arXiv:2606. 08543v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves large language model reasoning but often suffers from rapid policy-entropy collapse, where the policy prematurely concentrates on narrow high-probability reasoning paths.
By Shumeng Yang, Yisu Liu, Jiayi Zheng, Zhaohui Yang, Linjing Li
arXiv:2606. 31575v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a powerful tool for propelling Large Language Models (LLMs) beyond imitation-based training towards more robust reasoning capabilities.
By Outongyi Lv, Yanzhao Zheng, Yuanwei Zhang, Zhenghao Huang, Xingjun Wang, Baohua Dong, Hangcheng Zhu, Yingda Chen
arXiv:2606. 19771v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced Large Language Model (LLM) reasoning; however, it faces a fundamental optimization instability: uniform token updates precipitate entropy collapse, leading to premature convergence to suboptimal strategies, whereas excessive Shannon Entropy maximization can cause entropy explosion, driving blind exploration toward incoherent reasoning chains.
By Xuanzhi Feng, Zhengyang Li, Zeyu Liu, Haoxi Li, Yuming Jiang, Bing Guo, Jingcai Guo, Jie Zhang, Song Guo
arXiv:2610.20497v1 Announce Type: new
Abstract: Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual...
By Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang, Delu Zeng, Jiahao Wang, Junmei Yang, Pan tiewen, Jingfeng Liu, Zhiheng Zhou, Bo Li, Min Chen