arXiv AI
Aug 3
PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization
arXiv:2605. 17877v2 Announce Type: replace Abstract: A significant hurdle for current LLMs is the execution of complex, multi-stage tasks.
Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization
The paper introduces the Implicit Prefix-Value Reward Model (IPVRM), which learns the probability of eventual correctness for each prefix directly from outcome labels, thereby aligning training targets with inference-time step signals via temporal-difference differences. IPVRM improves step-verification F1 on ProcessBench. Additionally, the authors propose Distribution-Level RL (DistRL), a policy optimization method that applies TD advantages to both sampled and high-probability tokens, offering dense counterfactual updates without extra rollouts, and show that DistRL consistently enhances downstream reasoning when combined with IPVRM.
Related stories
arXiv Machine Learning
Jun 9
The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning
arXiv:2606. 09078v1 Announce Type: new Abstract: Process Reward Models (PRMs) improve credit assignment for reasoning by providing step-level feedback.
arXiv AI
Sep 30
Inducing Process Supervision from Outcome-Only Reinforcement Learning
arXiv:2609.36641v1 Announce Type: cross Abstract: Process reward models (PRMs) have become a key component for LLMs, as their step-level feedback supports both post-training and test-time reasoning....
arXiv AI
Oct 1
Learning Process Rewards via Reasoning State Propagation
The paper introduces Reasoning State Propagation (RSP), a method that models each reasoning prefix with a binary validity state and learns transitions between successive states. RSP predicts break and repair probabilities to connect intermediate reasoning states to the final outcome, enabling outcome supervision to guide learning of earlier steps. Experiments on reasoning search, response selection, and reinforcement learning show RSP consistently outperforms existing Process Reward Models, achieving notable gains over Qwen2.5-Math-PRM.
arXiv Machine Learning
Oct 22
EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping
EPIC introduces a new PPO-style clipping method that operates at the prefix level rather than token level, using the exact reverse-KL coordinate to regulate policy drift during autoregressive generation. By reformulating generation as a finite-horizon Markov decision process, the authors derive a performance-difference identity that identifies the cumulative likelihood ratio of the generated prefix as the exact importance weight for each policy-improvement step. Experiments on off‑policy reinforcement learning with a sparse mixture‑of‑experts language model demonstrate that EPIC consistently outperforms standard and sequence‑level clipping baselines on mathematical‑reasoning benchmarks while remaining competitive with routing‑replay alternatives.
arXiv AI
Sep 30
Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
arXiv:2609.37119v1 Announce Type: cross Abstract: Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instabilit...
arXiv AI
Oct 15
Unlocking Token Rewards via Training-Free Reward Attribution
arXiv:2610.18704v1 Announce Type: new Abstract: In this paper, we propose an extremely efficient, training-free method to extract token-level reward signals directly from an existing deep reward mode...
arXiv AI
Oct 21
TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment
TrajEvolve introduces a trajectory evolution paradigm with hindsight credit assignment to improve reinforcement learning for large language models on long-horizon tasks. It refines low-quality steps in historical trajectories to expose the policy to high-quality compositions that are rarely sampled, and reformulates step-level continuous credit assignment as a binary classification problem to identify redundant steps. Experiments on two datasets show that this approach outperforms existing methods.
arXiv AI
Aug 10
Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents
arXiv:2608. 06861v1 Announce Type: new Abstract: Training large language model agents in long-horizon environments requires assigning credit from sparse terminal outcomes to individual actions.
arXiv AI
Sep 1
ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth
arXiv:2512.03244v2 Announce Type: replace-cross Abstract: Training process reward models (PRMs) requires step-level correctness labels, obtained either through expensive human annotation or by relyin...
arXiv AI
Sep 28
Stepwise Intrinsic Rewards for Reasoning in Large Language Models
The paper introduces Stepwise Marginal Information Gain (MIG), an intrinsic process reward that evaluates how each reasoning step of a large language model (LLM) or vision-language model (VLM) improves the likelihood of the reference answer. MIG rewards only new likelihood maxima, preventing duplicate credit, and is combined with outcome, format, and self‑distillation objectives to guide training. Experiments on eight benchmarks show that this method outperforms outcome‑only reinforcement learning and improves accuracy by up to 4.8 points over binary‑reward training, including a 12.6‑point gain on MathVerse and a 12.9‑point advantage on vision‑language transfer at 7B parameters.
Hugging Face Trending Papers
Oct 21
EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping
Reinforcement learning (RL) based on PPO-style clipping has become an effective approach to improving the reasoning capabilities of large language models (LLMs). However, standard token-level clipping...