Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization

The paper introduces the Implicit Prefix-Value Reward Model (IPVRM), which learns the probability of eventual correctness for each prefix directly from outcome labels, thereby aligning training targets with inference-time step signals via temporal-difference differences. IPVRM improves step-verification F1 on ProcessBench. Additionally, the authors propose Distribution-Level RL (DistRL), a policy optimization method that applies TD advantages to both sampled and high-probability tokens, offering dense counterfactual updates without extra rollouts, and show that DistRL consistently enhances downstream reasoning when combined with IPVRM.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• reinforcement-learning