The paper investigates fine‑tuning reinforcement learning agents when offline data cannot be retained. It shows that continuing online behavioral cloning after the offline dataset disappears erodes the pretrained policy, leading to representational collapse and loss of action diversity. By freezing the expressive prior and stopping online behavioral fitting, the authors achieve better performance across several algorithms on seven OGBench domains.
arXiv:2608.20909v1 Announce Type: new Abstract: Offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions. This coupled l...
The paper introduces Bidirectional Behavior Prior Distillation (B2PD), a method that uses action‑value priors to train a conditional variational autoencoder for generating high‑value behavior support. These expert behavior priors are then distilled into the online reinforcement learning agent, reducing inefficient exploration and stabilizing policy updates. Experiments on state‑ and pixel‑based tasks show that B2PD improves sample efficiency while maintaining stable learning dynamics.
arXiv:2510. 01460v4 Announce Type: replace-cross Abstract: Offline-to-online reinforcement learning (RL) has emerged as a practical paradigm that leverages offline datasets for pretraining and online interactions for fine-tuning.
The paper introduces a minimal‑data adaptation regime where a pretrained robot policy learns a new task from a single demonstration followed by autonomous online interaction. It shows that while zero‑shot performance is poor, one demonstration can guide the policy toward coarse task‑following, and identifies key design principles—pretrained VLA representations for sample‑efficient value learning, residual policy learning for corrective behavior, and data‑balancing to avoid exploration collapse. The proposed MiDAS method, combining few‑demonstration behavior cloning with value‑based residual learning over frozen VLA representations, significantly boosts task success on LIBERO, RoboCasa, and a bimanual YAM platform after only a few hours of online interaction.
arXiv:2606. 25527v1 Announce Type: new Abstract: Online reinforcement learning (RL) agents increasingly depend on knowledge acquired offline to achieve practical efficiency.
arXiv:2502. 19544v3 Announce Type: replace Abstract: Leveraging offline data is a promising way to improve the sample efficiency of online reinforcement learning (RL).
arXiv:2602. 17632v3 Announce Type: replace-cross Abstract: Modern offline Reinforcement Learning (RL) methods find performant actor-critics, however, fine-tuning these actor-critics online with value-based RL algorithms typically causes immediate drops in performance.
arXiv:2609.35954v1 Announce Type: cross Abstract: Large language model post-training generates self-generated rollouts through reinforcement learning and on-policy distillation, yet this experience i...
Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it should, but recent results show that online RL with a randomly-initialized Q-function can result in highly performant and reliable policies without needing to pretrain the Q-function.
arXiv:2608. 01205v1 Announce Type: new Abstract: Recent offline reinforcement learning methods increasingly rely on expressive generative policies and specialized value-guidance mechanisms.
arXiv:2607. 27203v1 Announce Type: new Abstract: Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too?
arXiv:2510.15047v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) as agents often fail to improve in new environments. We identify and characterize a failure mode we call explora...