← Back to all news
arXiv Machine Learning By Dong Shu, Denghui Zhang, Jessica Hullman

Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training

Read the original on arXiv Machine Learning →

The paper introduces Influence‑Guided PPO (I‑PPO), a framework that incorporates data attribution into the Proximal Policy Optimization (PPO) post‑training loop for large language models. I‑PPO computes an influence score for each episode via a gradient‑based approximation and removes episodes that are anti‑aligned with a validation gradient, thereby filtering out noisy or unfaithful reasoning. Experiments show that this filtering consistently outperforms both supervised fine‑tuning (SFT) and standard PPO, acting as an intrinsic early‑stopping mechanism that speeds up training and reduces unfaithful chain‑of‑thought reasoning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Aug 25

Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization

The paper introduces the Implicit Prefix-Value Reward Model (IPVRM), which learns the probability of eventual correctness for each prefix directly from outcome labels, thereby aligning training targets with inference-time step signals via temporal-difference differences. IPVRM improves step-verification F1 on ProcessBench. Additionally, the authors propose Distribution-Level RL (DistRL), a policy optimization method that applies TD advantages to both sampled and high-probability tokens, offering dense counterfactual updates without extra rollouts, and show that DistRL consistently enhances downstream reasoning when combined with IPVRM.

By Shiping Gao, Hongzhan Chen, Xiaojun Quan, Qifan Wang, Lifu Huang
reinforcement-learning
More like this →
arXiv AI
Oct 15

Learning with a Single Rollout via Pass@k Policy Optimization

The paper introduces Single‑Rollout Pass@k Policy Optimization (SR‑PPO), a reinforcement learning method that uses only one rollout per prompt to train language‑model agents. SR‑PPO combines Monte Carlo token‑level advantage estimation with Pass@k gradient reweighting and an online critic for baseline and weight estimation. Experiments show that SR‑PPO achieves stable learning and performance comparable to traditional Group Relative Policy Optimization (GRPO) on reasoning and logic benchmarks, even when off‑policy drift degrades standard Pass@1 PPO.

By Fengdi Che, Yang Liu, Lei Yu, Meng Cao, Tong Che, Rupam Mahmood, Dale Schuurmans
agentsreinforcement-learningbenchmarks
More like this →