arXiv AI August 25, 2026 By Penghui Qi, Xiangxin Zhou, Wee Sun Lee

How to Train a Critic Stably and Efficiently

The paper introduces Best‑Practice Critic Optimization (BPCO), a stable and efficient recipe for training a critic in reinforcement learning for large language models. BPCO combines DPPO, bounded value predictions, Monte Carlo targets, unnormalized policy advantages, and length‑adaptive advantage estimation, allowing the critic to be conditioned on hidden reward information. Experiments on mathematical reasoning tasks with models from 1.5B to 30B parameters show that BPCO consistently outperforms a strong critic‑based baseline and matches or exceeds group‑based methods while sampling only one response per prompt.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

4d ago

Best Practice Critic Optimization

arXiv:2608.23566v2 Announce Type: replace-cross Abstract: Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for...

By Penghui Qi, Xiangxin Zhou, Wee Sun Lee

llms reinforcement-learning

arXiv AI

Jun 30

BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

arXiv:2606. 28707v1 Announce Type: new Abstract: Critic-free reinforcement learning with verifiable rewards (RLVR), exemplified by Group Relative Policy Optimization (GRPO), avoids training a value function (critic) and reduces memory and compute overhead relative to critic-based PPO pipelines for aligning large language models.

By Yupeng Chang, Yuan Wu, Yi Chang

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 19

VIMPO: Value-Implicit Policy Optimization for LLMs

arXiv:2606. 20008v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards has become a central tool for improving the reasoning ability of large language models, but current methods face a trade-off between simplicity and credit assignment.

By Zhewei Kang, Aosong Feng, Sergey Levine, Dawn Song, Xuandong Zhao

llms reinforcement-learning benchmarks

arXiv Machine Learning

Aug 18

Le Critique: Privileged Value Functions for LLM Reinforcement Learning

arXiv:2608. 16739v1 Announce Type: new Abstract: Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy.

By Siddarth Venkatraman, Matthieu Dinot, Laurence Aitchison

llms reinforcement-learning efficiency

arXiv Machine Learning

Jun 2

Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation

arXiv:2605. 21125v2 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO), a prominent algorithm within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, has achieved strong results in improving the reasoning capabilities of large language models (LLMs).

By Xixiang He, Qiyao Sun, Ao Cheng, Xingming Li, Xuanyu Ji, Hailun Lu, Runke Huang, Qingyong Hu

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 4

Policy Improvement Reinforcement Learning

arXiv:2604. 00860v3 Announce Type: replace Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become a central post-training paradigm for improving the reasoning capabilities of large language models.

By Huaiyang Wang, Xiaojie Li, Deqing Wang, Haoyi Zhou, Zixuan Huang, Yaodong Yang, Jianxin Li, Yikun Ban

llms reinforcement-learning benchmarks

One email a morning, machine-written

Related stories

Best Practice Critic Optimization

BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

VIMPO: Value-Implicit Policy Optimization for LLMs

Le Critique: Privileged Value Functions for LLM Reinforcement Learning

Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation

Policy Improvement Reinforcement Learning