arXiv:2509. 25004v2 Announce Type: replace Abstract: Online reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning abilities of large language models, but most methods still optimize reasoning trajectories over the static problem set, wasting rollout budget on solved or overly difficult problems.
By Shijie Zhang, Zheng Xiao, Shiyu Liu, Guohao Sun, Kevin Zhang, Xiang Guo, Rujun Guo, Shaoyu Liu, Wangxiao Zhao, Guanjun Jiang
AdaC-GRPO introduces an adaptive curriculum learning framework for mastering long-horizon visual Sudoku puzzles, a multimodal task that challenges large language models with multi-step deductive reasoning. The authors present FG-Sudoku, a fine-grained benchmark that categorizes puzzles by grid size and required deductive steps, revealing that frontier multimodal large language models (MLLMs) struggle with complex long-range dependencies. AdaC-GRPO dynamically constructs training batches by tracking the model’s Zone of Proximal Development through online rollout accuracy, enabling Qwen2.5-VL-7B to close a significant portion of the accuracy gap to Gemini-2.5-Pro and demonstrating generalizability across other models.
By Sida Li, Yuanxin Liu, Linli Yao, Shicheng Li, Xu Sun, Haoli Bai, Xiaohui Li
Goldilocks RL proposes an adaptive data‑selection strategy for reinforcement learning in language models, using a Selector network to predict reward variability and prioritize questions that are neither too easy nor too hard. By continuously adapting to the model’s evolving abilities, Goldilocks improves training efficiency on large‑scale reasoning datasets, achieving up to 78% fewer optimization steps compared to standard GRPO. The approach addresses the sample‑inefficiency problem of sparse rewards in reasoning tasks.
By Ilia Mahrooghi, Aryo Lotfi, Emmanuel Abbe
arXiv:2606. 07950v1 Announce Type: new Abstract: RL with verifiable rewards can substantially improve LLM reasoning, yet standard GRPO-style training often treats easy, hard, and learnable questions alike through uniform sampling and weighting, leading to inefficient compute allocation.
By Zhanke Zhou, Xiangyu Lu, Chentao Cao, Brando Miranda, Tongliang Liu, Bo Han, Sanmi Koyejo
The paper introduces Policy-Compatible Reinforcement Learning (PICO), a method that addresses the learning cliff in reinforcement learning with verifiable rewards (RLVR) by performing distribution-constrained joint decoding. PICO guides token-level decoding with expert data while preserving the model’s token probabilities, and uses a dynamic top‑k mechanism to keep sampling within a finite divergence budget. The resulting trajectories are verified and policy‑compatible, and are jointly optimized with on‑policy rollouts using a mixed‑behavior objective that mitigates distribution mismatch via importance sampling, leading to consistent performance gains over standard RLVR and prior off‑policy baselines on diverse reasoning datasets and agentic benchmarks.
By Zehao Wang, Hanzhu Chen, Runquan Gui, Zhe Wang, Lin Yang
arXiv:2609.39321v1 Announce Type: cross
Abstract: Group Relative Policy Optimization (GRPO) has emerged as a memory-efficient reinforcement fine-tuning (RFT) technique for reasoning-intensive tasks....
By Rajat Ghosh, Vaishnavi Bhargava, Henry Wong, Aryan Singhal, Debojyoti Dutta