The paper introduces Multi‑Fidelity Policy Gradient (MFPG) methods that combine scarce, high‑fidelity (HF) reinforcement‑learning data with abundant, low‑fidelity (LF) data to stabilize training. Unlike prior approaches that directly optimize biased LF objectives, MFPG uses LF data only to construct a control variate that reduces variance without biasing the policy gradient. The authors extend MFPG to modern actor‑critic algorithms, notably a budget‑aware MFPG‑PPO that preserves cross‑fidelity correlation, monitors estimator uncertainty, and allocates a fixed sampling budget between HF and LF sources. Experiments on simulated robot locomotion and a physical Franka arm show that MFPG‑PPO consistently outperforms standard PPO trained on HF data alone and matches performance achieved with 16× more HF data on the hardest tasks, while enabling stable learning with as few as four real‑robot episodes per update.
By Xinjie Liu, Ruihan Zhao, Anirban Chaudhuri, Cyrus Neary, Ufuk Topcu, David Fridovich-Keil
arXiv:2605. 03065v2 Announce Type: replace Abstract: Generative control policies (GCPs), such as diffusion- and flow-based control policies, have emerged as effective parameterizations for robot learning.
By Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal, Shashwat Saxena, Jesse Zhang, Giri Anantharaman, Cleah Winston, Chaoyi Pan, Douglas Chen, Nai-Chieh Huang, Zeynep Temel, Oliver Kroemer, Sergey Levine, Abhishek Gupta, Hongkai Dai, Paarth Shah, Max Simchowitz
arXiv:2607. 26509v1 Announce Type: new Abstract: Deep off-policy reinforcement learning algorithms for continuous control typically rely on neural value function approximation to guide policy improvement.
By Gong Gao, Xiao Lai, Ziqi Xie, Guojie Chen, Xianhui Liu, Weidong Zhao
The paper investigates how non‑monotonic performance dynamics in single‑task reinforcement learning affect policy selection. By introducing post‑hoc diagnostic metrics, the authors evaluate common selection strategies on continuous‑control benchmarks and find that typical practices often miss the best policy, leading to suboptimal performance and wasted computation. They propose a validation‑based selection method that better approximates the best visited policy while improving efficiency.
By Judith Echevarrieta, Etor Arza, Aritz P\'erez, Josu Ceberio
arXiv:2606. 05888v1 Announce Type: new Abstract: Retry-based objectives such as pass@K and max@K optimize the best return obtained from multiple sampled trajectories, and recent work has shown that they can promote exploration without explicit exploration bonuses.
By Soichiro Nishimori, Paavo Parmas
arXiv:2606. 11087v1 Announce Type: cross Abstract: Expressive continuous control policies, such as diffusion and flow models, form the backbone of recent advances in scaling imitation learning for simulated and real robot control.
By Zhiyuan Zhou, Andy Peng, Charles Xu, Qiyang Li, Tobias Springenberg, Kevin Frans, Sergey Levine
arXiv:2607. 21302v1 Announce Type: new Abstract: Behavior prior reinforcement learning (BPRL) has emerged as a promising paradigm to improve sample efficiency in online reinforcement learning (RL) by leveraging policy priors derived from offline demonstrations.
By Gong Gao, Weidong Zhao, Xianhui Liu, Ning Jia
arXiv:2602. 05379v2 Announce Type: replace-cross Abstract: Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data to improve sample efficiency and accelerate policy optimization.
By Hua Zheng, Wei Xie, M. Ben Feng, Keilung Choy
The paper introduces Bidirectional Behavior Prior Distillation (B2PD), a method that uses action‑value priors to train a conditional variational autoencoder for generating high‑value behavior support. These expert behavior priors are then distilled into the online reinforcement learning agent, reducing inefficient exploration and stabilizing policy updates. Experiments on state‑ and pixel‑based tasks show that B2PD improves sample efficiency while maintaining stable learning dynamics.
By Gong Gao, Xiao Lai, Jiaji Shen, Ning Jia, Xianhui Liu, Weidong Zhao
VaLoP introduces a new method for reward-guided diffusion called Value Gradient Lookahead and Pullback, which evaluates the value function at a less noisy state and pulls its gradient back to the current state to reduce posterior covariance errors. The approach unifies two common gradient estimators and adds a correction term computed with a single vector-Jacobian product, while VaLoP‑Fast amortizes this correction with a lightweight head trained on a single reward. Experiments across multiple models, rewards, and datasets show that both variants consistently outperform existing value‑gradient guidance, and VaLoP‑Fast achieves the highest success rates on robot policy benchmarks such as LIBERO and RoboCasa.
By Kwanyoung Kim
arXiv:2610.19783v1 Announce Type: new
Abstract: Reinforcement learning (RL) holds the promise of learning a policy that can outperform what is shown in the data. However, in practice, such systems su...
By Zhiyuan Zhou, Sergey Levine
arXiv:2606. 14801v1 Announce Type: cross Abstract: Flow-matching and diffusion policies are expressive action generators, but optimizing them with temporal-difference reinforcement learning (RL) remains difficult.
By Yifan Ruan, Chenyang Cao, Andreas Burger, Ali Pesaranghader, Kaveh Kamali, Jaehong Kim, Nandita Vijaykumar, Alan Aspuru-Guzik, Igor Gilitschenski, Nicholas Rhinehart