SetGRPO is a new method for calibrating grader‑induced policy optimization in rubric‑based reinforcement learning. It introduces three modules—Joint Reward Set Calibration, Geometry‑Guided Safe Recovery, and Selective Risk‑Gated Optimization—to construct uncertainty sets for training coefficients, map them into safe update geometry, and control direction‑reversal risk. Experiments on ten open‑ended benchmarks show that SetGRPO outperforms other training methods on both Qwen3.5 and Llama3.2 models.
arXiv:2606. 06058v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards is ideal for multi-constraint instruction following, yet standard group-relative policy optimization (GRPO) becomes unstable under discrete, low-dispersion rewards, where within-group reward distributions are frequently homogeneous.
arXiv:2606. 28707v1 Announce Type: new Abstract: Critic-free reinforcement learning with verifiable rewards (RLVR), exemplified by Group Relative Policy Optimization (GRPO), avoids training a value function (critic) and reduces memory and compute overhead relative to critic-based PPO pipelines for aligning large language models.
The paper introduces Best‑Practice Critic Optimization (BPCO), a stable and efficient recipe for training a critic in reinforcement learning for large language models. BPCO combines DPPO, bounded value predictions, Monte Carlo targets, unnormalized policy advantages, and length‑adaptive advantage estimation, allowing the critic to be conditioned on hidden reward information. Experiments on mathematical reasoning tasks with models from 1.5B to 30B parameters show that BPCO consistently outperforms a strong critic‑based baseline and matches or exceeds group‑based methods while sampling only one response per prompt.
arXiv:2610.15840v1 Announce Type: cross Abstract: Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assign...
arXiv:2604. 00860v3 Announce Type: replace Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become a central post-training paradigm for improving the reasoning capabilities of large language models.
arXiv:2606. 04807v1 Announce Type: new Abstract: Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape.
arXiv:2608.23566v2 Announce Type: replace-cross Abstract: Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for...
arXiv:2607. 18470v1 Announce Type: cross Abstract: Group Relative Policy Optimization (GRPO) has shown strong effectiveness in reinforcement learning from verifiable feedback, where sampled rollouts can be compared within a group using task-provided correctness signals.
arXiv:2609.36178v1 Announce Type: cross Abstract: Group Relative Policy Optimization (GRPO) has become a promising approach for training large language model agents. However, its uniform assignment o...
The paper introduces Follow the Winners (FTW), a critic‑free policy‑learning algorithm that adapts the cross‑entropy method for reinforcement fine‑tuning of agentic large language models. FTW replaces group rollouts with an ordinal filter on replay‑buffer samples, achieving polynomial concentration in the order statistic of returns and offering a bounded risk‑seeking offset that balances variance reduction. Experiments on Sokoban and Search‑R1 show that FTW matches the performance of GRPO and PPO while reducing reliance on value models or repeated rollouts.
arXiv:2609.39757v1 Announce Type: cross Abstract: Black-box distillation is a practical route for transferring capabilities from API-accessible large language models that expose only text outputs int...
arXiv:2609.40360v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions...