arXiv AI
Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 19
VIMPO: Value-Implicit Policy Optimization for LLMs
arXiv:2606. 20008v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards has become a central tool for improving the reasoning ability of large language models, but current methods face a trade-off between simplicity and credit assignment.
arXiv Computation and Language
Sep 10
Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning
arXiv:2604.10701v2 Announce Type: replace-cross Abstract: Credit assignment is a central challenge in reinforcement learning (RL). Classical actor-critic methods address this challenge through fine-g...
arXiv AI
1d ago
Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States
The paper introduces POISE, a reinforcement learning algorithm that uses a model’s internal states as a value estimator to reduce variance in reinforcement learning with verifiable rewards (RLVR). By employing a lightweight probe that reads internal signals during the forward pass, POISE predicts baselines online and uses a cross‑rollout construction to keep gradients unbiased. Experiments on Qwen3‑4B and OLMo3‑7B‑Instruct‑DPO across six domains show POISE outperforms existing RLVR baselines, offering more stable training and a value model that generalizes across tasks and scales with the policy.
arXiv Machine Learning
Jul 21
Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR
arXiv:2509. 02522v3 Announce Type: replace-cross Abstract: Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have empowered large language models (LLMs) to tackle challenging reasoning tasks such as mathematics and programming, however existing RLVR methods often suffer from sparse reward signals and unstable policy gradient updates inherent to RL-based approaches.
arXiv Machine Learning
Aug 18
Le Critique: Privileged Value Functions for LLM Reinforcement Learning
arXiv:2608. 16739v1 Announce Type: new Abstract: Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy.
arXiv AI
3d ago
SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
SIPO (Self‑Instructing Policy Optimization) unifies reinforcement learning with on‑policy self‑distillation by using a contrastive self‑teacher to generate token‑level credit signals. The method samples multiple rollouts per prompt, pairs each with a reference answer and its mistakes, and uses the difference in teacher log‑probabilities to provide dense feedback while still respecting the overall task reward. Experiments on reasoning and code‑generation benchmarks show that SIPO outperforms both RLVR and OPSD baselines without requiring an external teacher or extra generation steps.