arXiv Machine Learning

Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

arXiv Machine Learning
Sep 21

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

The paper introduces PARTS, a real‑world subtask reinforcement learning framework that fine‑tunes a pretrained robot policy by focusing on critical bottleneck subtasks while keeping the base policy frozen. It uses agent‑generated selectors and success verifiers to provide local rewards, enabling learning even when full‑task successes are rare. Experiments on bimanual YAM and single‑arm Franka robots show that PARTS raises complete‑task success from 32% to 61% and from 50% to 95%, respectively, with only tens of minutes of real‑world RL rollouts and minimal human intervention.

By Sichang Su, Benjamin Yang, Zhiyun Deng, Boyuan Liang, Yip Fun Yeung, Zelin Wang, Lingfeng Sun
arXiv AI
Sep 24

BEE: Intervention-Adaptive Real-World Reinforcement Learning with Vision-Language-Action Models

arXiv:2609.27450v1 Announce Type: cross Abstract: Vision-language-action (VLA) models handle long-horizon manipulation, yet success hinges on a few precision-critical phases where millimeter-scale er...

By Weihui Zhao, Xiaohan Yan, Zunian Wan, Xuan Du, Zhaozhan Chi, Jianbo Mao, Ruipu Wu, Rushuai Yang, Houlin Li, Shukai Yang, Jing Wu, Yuxiang Yan, Yongcheng Liu, Chuankang Li, Guanghui Ren, Wei Shan, Maoqing Yao
arXiv AI
Aug 13

TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning

arXiv:2605. 12236v2 Announce Type: replace-cross Abstract: Fine-tuning pre-trained robot policies with reinforcement learning (RL) often inherits the bottlenecks introduced by pre-training with behavioral cloning (BC), which produces narrow action distributions that lack the coverage necessary for downstream exploration.

By Matthew M. Hong, Jesse Zhang, Anusha Nagabandi, Abhishek Gupta
arXiv Machine Learning
3d ago

What Does It Take For A Generalist Robot Policy To Adapt Using Minimal Data?

The paper introduces a minimal‑data adaptation regime where a pretrained robot policy learns a new task from a single demonstration followed by autonomous online interaction. It shows that while zero‑shot performance is poor, one demonstration can guide the policy toward coarse task‑following, and identifies key design principles—pretrained VLA representations for sample‑efficient value learning, residual policy learning for corrective behavior, and data‑balancing to avoid exploration collapse. The proposed MiDAS method, combining few‑demonstration behavior cloning with value‑based residual learning over frozen VLA representations, significantly boosts task success on LIBERO, RoboCasa, and a bimanual YAM platform after only a few hours of online interaction.

By Shreyas Kowshik, Sreyas Venkataraman, Leo Wang, Niharika Pant, Max Simchowitz, Aviral Kumar
arXiv AI
Oct 9

Agentic Critical Training

Agentic Critical Training (ACT) is a reinforcement learning approach that trains language‑model agents to judge actions directly by pairing expert actions with alternatives and rewarding correct selection. Unlike imitation learning or self‑reflection, ACT does not require reference rationales and can be reused across model sizes. Experiments on Qwen3‑8B and Olmo‑3‑7B‑Instruct show that ACT improves performance on ALFWorld, WebShop, ScienceWorld, and outperforms supervised reflection baselines, with significant gains over IL and IL→RL baselines.

By Weize Liu, Minghui Liu, Sy-Tuyen Ho, Yongkyun Lee, Andrew Adams Schoen, Souradip Chakraborty, Xiyao Wang, Furong Huang
arXiv Machine Learning
Jun 11

OGPO: Sample Efficient Full-Finetuning of Generative Control Policies

arXiv:2605. 03065v2 Announce Type: replace Abstract: Generative control policies (GCPs), such as diffusion- and flow-based control policies, have emerged as effective parameterizations for robot learning.

By Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal, Shashwat Saxena, Jesse Zhang, Giri Anantharaman, Cleah Winston, Chaoyi Pan, Douglas Chen, Nai-Chieh Huang, Zeynep Temel, Oliver Kroemer, Sergey Levine, Abhishek Gupta, Hongkai Dai, Paarth Shah, Max Simchowitz
arXiv Machine Learning
Jun 2

Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards

arXiv:2606. 02194v1 Announce Type: new Abstract: Distilling expert demonstration data into large generative models using behavioral cloning is a scalable approach to learning capable policies for robotic control, particularly for dexterous manipulation.

By Christian Scherer, Joe Watson, Theo Gruner, Daniel Palenicek, Ingmar Posner, Jan Peters