arXiv Machine Learning October 3, 2026 By Alessandro Montenegro, Riccardo Venturelli, Marco Mussi, Matteo Papini, Alberto Maria Metelli

Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?

Read the original on arXiv Machine Learning →

The paper investigates how reusing past samples can improve the sample efficiency of Proximal Policy Optimization (PPO). Two variants, wPPO-U and wPPO-BH, are introduced within a multiple importance weighting framework, each reusing data from recent iterations while preserving core PPO mechanics. The authors derive theoretical policy improvement bounds for both variants and empirically evaluate their impact on continuous control tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

reinforcement-learning

arXiv Machine Learning

Jun 16

Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning

arXiv:2606. 15333v1 Announce Type: cross Abstract: LLM unlearning has emerged as a cost-effective alternative to full retraining for removing hazardous knowledge from pretrained models while preserving general utility.

By Zirui Pang, Chenlong Zhang, Haosheng Tan, Zhuoran Jin, Jiaheng Wei, Zixin Zhong

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 4

Reusing Trajectories in Policy Gradients Enables Fast Convergence

arXiv:2506. 06178v3 Announce Type: replace Abstract: Policy gradient (PG) methods are a class of effective reinforcement learning algorithms, particularly when dealing with continuous control problems.

By Alessandro Montenegro, Federico Mansutti, Marco Mussi, Matteo Papini, Alberto Maria Metelli

reinforcement-learning benchmarks

arXiv AI

4d ago

Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO

The paper introduces FastRL, a reinforcement learning framework designed to enhance the efficiency of Group Relative Policy Optimization (GRPO) and its variants. FastRL employs an advantage-aware pruning strategy that retains high-advantage trajectories while preserving gradient diversity, and an adaptive rollout sampling mechanism that adjusts sampling scale during training based on historical pruning data. Experiments show that FastRL can be integrated into GRPO, DAPO, and GSPO, yielding a 2.07× speedup on Geometry3K and GeoQA8K-R1V and a 1.64% accuracy improvement on visual reasoning benchmarks.

By Jiahua Yang, Zhiwei Yang, Xianpeng Zhang, Dongyu Chen, Xing Chen, Tianhuang Su, Haonan Lu, Quanlong Guan, Kai Tang, Chuangchuang Wang

reinforcement-learning efficiency benchmarks

arXiv AI

2d ago

Finetuning with Sampling: SFT Learns Better Than You Think

arXiv:2610.02140v1 Announce Type: cross Abstract: Introducing new capabilities to frontier models has long been the goal of posttraining, which predominantly employs supervised finetuning (SFT) and r...

By Aayush Karan, Sitan Chen, Yilun Du

reinforcement-learning fine-tuning

arXiv Machine Learning

Aug 14

Variance Reduction Based Experience Replay for Policy Optimization

arXiv:2602. 05379v2 Announce Type: replace-cross Abstract: Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data to improve sample efficiency and accelerate policy optimization.

By Hua Zheng, Wei Xie, M. Ben Feng, Keilung Choy

reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 21

Rethinking the Suitability of Reinforcement Learning Algorithms Under Practical Transfer Constraints

arXiv:2607. 17326v1 Announce Type: new Abstract: Transfer-oriented reinforcement learning requires evaluating algorithms along dimensions that go beyond standard sample efficiency.

By Hany Hamed, Abhishek Naik, Colin Bellinger, A. Rupam Mahmood

reinforcement-learning

One email a morning, machine-written

Related stories

Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning

Reusing Trajectories in Policy Gradients Enables Fast Convergence

Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO

Finetuning with Sampling: SFT Learns Better Than You Think

Variance Reduction Based Experience Replay for Policy Optimization

Rethinking the Suitability of Reinforcement Learning Algorithms Under Practical Transfer Constraints