← Back to all news
arXiv Computation and Language By Shiping Gao, Hongzhan Chen, Xiaojun Quan, Qifan Wang, Lifu Huang

Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization

Read the original on arXiv Computation and Language →

The paper introduces the Implicit Prefix-Value Reward Model (IPVRM), which learns the probability of eventual correctness for each prefix directly from outcome labels, thereby aligning training targets with inference-time step signals via temporal-difference differences. IPVRM improves step-verification F1 on ProcessBench. Additionally, the authors propose Distribution-Level RL (DistRL), a policy optimization method that applies TD advantages to both sampled and high-probability tokens, offering dense counterfactual updates without extra rollouts, and show that DistRL consistently enhances downstream reasoning when combined with IPVRM.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Oct 1

Learning Process Rewards via Reasoning State Propagation

The paper introduces Reasoning State Propagation (RSP), a method that models each reasoning prefix with a binary validity state and learns transitions between successive states. RSP predicts break and repair probabilities to connect intermediate reasoning states to the final outcome, enabling outcome supervision to guide learning of earlier steps. Experiments on reasoning search, response selection, and reinforcement learning show RSP consistently outperforms existing Process Reward Models, achieving notable gains over Qwen2.5-Math-PRM.

By Kai Gan, Zi-Hao Zhou, Bo Ye, Jian Zhao, Min-Ling Zhang, Tong Wei
reinforcement-learning
More like this →
arXiv Machine Learning
Oct 22

EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping

EPIC introduces a new PPO-style clipping method that operates at the prefix level rather than token level, using the exact reverse-KL coordinate to regulate policy drift during autoregressive generation. By reformulating generation as a finite-horizon Markov decision process, the authors derive a performance-difference identity that identifies the cumulative likelihood ratio of the generated prefix as the exact importance weight for each policy-improvement step. Experiments on off‑policy reinforcement learning with a sparse mixture‑of‑experts language model demonstrate that EPIC consistently outperforms standard and sequence‑level clipping baselines on mathematical‑reasoning benchmarks while remaining competitive with routing‑replay alternatives.

By Chengxi Li, Hanlin Wu, Zhicheng Cai, Huan-ang Gao, Wei-Ying Ma, Jingjing Liu, Hao Zhou
llmsreinforcement-learningbenchmarks
More like this →