The paper introduces Multi-step Proximal Policy Improvement (MPI), a method that refines offline reinforcement learning policies through sequential re-centered proximal steps. By viewing policies as a probability manifold, MPI interprets a wide range of offline actor objectives as a single proximal policy improvement step and extends this to multiple steps for controlled policy improvement beyond the behavior distribution. Experiments on D4RL benchmarks demonstrate that a few MPI refinements enhance strong offline baselines such as TD3+BC, ReBRAC, and IQL, while diagnostics clarify the benefits of re-centered refinement over fixed-objective scheduling and highlight critic error limitations.
By Soohyun Choi, Seonvin Cho, Songnam Hong
The paper introduces Multi-step Proximal Policy Improvement (MPI), a method that refines offline reinforcement learning policies through sequential re-centered proximal steps. By modeling policies as a probability manifold, MPI interprets a wide range of offline actor objectives as a single proximal policy improvement step and extends this to multiple steps for controlled policy improvement beyond the behavior distribution. Experiments on D4RL benchmarks demonstrate that a few MPI refinements enhance strong offline baselines such as TD3+BC, ReBRAC, and IQL across many tasks, while diagnostics clarify the benefits of re-centered refinement over fixed-objective scheduling and highlight critic error limitations.
arXiv:2609.40149v1 Announce Type: new
Abstract: Policy regularization in offline reinforcement learning balances policy improvement against reliance on uncertain value estimates. This balance can dif...
By Seonvin Cho, Soohyun Choi, Songnam Hong
arXiv:2608. 03483v1 Announce Type: cross Abstract: Existing chunk-based Vision-Language-Action (VLA) models execute a fixed number of actions (i.
By Weichen Xu, Zhenhua Liu, Lin Luo, Yaobo Liang, Chengtang Yao, Qingyu Mei, Jian Cao, Xixin Cao, Xing Zhang, Jiaolong Yang, Baining Guo
arXiv:2605. 03065v2 Announce Type: replace Abstract: Generative control policies (GCPs), such as diffusion- and flow-based control policies, have emerged as effective parameterizations for robot learning.
By Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal, Shashwat Saxena, Jesse Zhang, Giri Anantharaman, Cleah Winston, Chaoyi Pan, Douglas Chen, Nai-Chieh Huang, Zeynep Temel, Oliver Kroemer, Sergey Levine, Abhishek Gupta, Hongkai Dai, Paarth Shah, Max Simchowitz
arXiv:2606. 05888v1 Announce Type: new Abstract: Retry-based objectives such as pass@K and max@K optimize the best return obtained from multiple sampled trajectories, and recent work has shown that they can promote exploration without explicit exploration bonuses.
By Soichiro Nishimori, Paavo Parmas
arXiv:2510. 03494v2 Announce Type: replace Abstract: We study finite-horizon offline reinforcement learning (RL) with function approximation for both policy evaluation and policy optimization.
By Volodymyr Tkachuk, Csaba Szepesv\'ari, Xiaoqi Tan
arXiv:2606. 31043v1 Announce Type: new Abstract: Residual reinforcement learning adapts a pretrained robot policy by learning an additive correction to its actions.
By Ethan Hirschowitz, Fabio Ramos
arXiv:2608.20909v1 Announce Type: new
Abstract: Offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions. This coupled l...
By Xuyao Lin, Yixiang Shan, Jinru Duan, Tao Yang, Xinyu Zhao, Runyu Lei, Yiming Zhao, Jiaxin Fan, Zongbao Feng, Peng Jia
arXiv:2603. 13707v3 Announce Type: replace-cross Abstract: Humanoid loco-manipulation requires coordinated task-space motion planning with stable loco-manipulation command tracking under complex robot-environment dynamics and long-horizon tasks.
By Zhaoyuan Gu, Yipu Chen, Zimeng Chai, Alfred Cueva, Thong Nguyen, Yifan Wu, Huishu Xue, Minji Kim, Isaac Legene, Fukang Liu, KyoungMok Kim, Ayan Barula, Yongxin Chen, Ye Zhao
arXiv:2609.21108v1 Announce Type: new
Abstract: Deep reinforcement learning (DRL) has achieved strong performance across a wide range of continuous-control problems. These continuous-control policies...
By Sachini Weerasekara, Sagar Kamarthi, Jacqueline Isaacs
arXiv:2605. 30719v2 Announce Type: replace-cross Abstract: We study when large language models (LLMs) can serve as effective black-box policy optimizers for reinforcement learning (RL) tasks, i.
By Stephane Hatgis-Kessell, Emma Brunskill