HorizonFlow is a hierarchical planner for offline goal-conditioned reinforcement learning that treats the planning horizon as an output rather than a fixed input. It uses a subgoal route planner and an action-prefix controller, both employing insertion-based generation and flow matching, to jointly generate continuous plan content and its length. The method leverages the partially generated plan to guide token insertion and to steer generation toward shorter plans, achieving superior performance on Maze2D, Multi2D, and OGBench benchmarks.
Recent advances in generative planning have made trajectory inpainting a promising approach to offline goal-conditioned reinforcement learning. However, these methods typically specify the planning ho...
arXiv:2608.29061v1 Announce Type: new Abstract: Offline goal-conditioned reinforcement learning (GCRL) aims to learn policies for reaching diverse goals entirely from fixed trajectory data. Long-hori...
arXiv:2607. 20834v1 Announce Type: new Abstract: Offline goal-conditioned reinforcement learning (RL) holds the promise of learning general-purpose policies from static datasets.
arXiv:2607. 17973v1 Announce Type: new Abstract: Latent world models have emerged as a powerful planning paradigm by learning action-conditioned predictive dynamics and using them as internal simulators to imagine and evaluate candidate action sequences.
Offline goal-conditioned reinforcement learning (RL) holds the promise of learning general-purpose policies from static datasets. However, scaling these methods to long-horizon tasks remains a challenge due to the curse of horizon, where value estimation errors can compound through long chains of bootstrapped Bellman backups.
The paper demonstrates that planners using frozen visual world models can achieve better control by changing the target used for action scoring. Instead of scoring actions solely by distance to the final goal image, the authors propose Anchored Planning, which retrieves a recorded trajectory segment that matches the current and goal observations and then scores actions toward an intermediate observation shortly after the segment’s start. Experiments on Cube, PushT, Reacher, and TwoRoom show that this intermediate-target approach outperforms the released LeWM planner on all long‑range tasks, while simple final‑goal search fails to achieve the same gains.
arXiv:2608. 14125v1 Announce Type: new Abstract: LeWM is a lightweight visual world model that learns latent dynamics end-to-end from pixels and ranks candidate action sequences by the distance between their predicted endpoints and the goal.
The paper introduces DCRL (Divide-and-Conquer RL), a method that recursively decomposes offline goal-conditioned reinforcement learning trajectories into a balanced binary tree. By training values from the leaves up to the root, DCRL avoids noisy max-based backups and reduces bootstrap depth from linear to logarithmic, thereby limiting error accumulation. Experiments on diverse goal-reaching tasks show that DCRL outperforms prior flat offline GCRL methods, achieving a higher average score on the most challenging long-horizon OGBench tasks.
arXiv:2607. 09336v1 Announce Type: cross Abstract: Diffusion-based trajectory planners have shown strong performance in offline reinforcement learning, but their iterative denoising process often incurs high inference cost.
The paper introduces the Dual-Latent World Model (Dual-WM), which separates local execution and long-range planning into distinct latent spaces and dynamics models. A new learning method, Long-Horizon Representation Learning with Weighted Rollout (LoRe), supervises predictions at both levels using exponential horizon weights. Experiments on five goal-conditioned visual control tasks show that Dual-WM improves success rates over strong baselines, especially at longer horizons.
Reinforced Planning with Latent World Models introduces RP1, a neural planner that learns to evaluate imagined outcomes via a critic and improve multi‑step plans through an optimizer trained offline on world‑model roll‑outs. Unlike existing planners that are hand‑designed or only inform policies, RP1 fully learns to refine plans and can be attached to any pretrained latent world model. In experiments on visual navigation, arm reaching, and robotic manipulation, RP1 outperforms hand‑designed search algorithms, achieving near‑perfect success while using 1,000× fewer roll‑outs and up to 67× faster inference.
Reinforced Planning with Latent World Models (RP1) is a novel method that learns to evaluate imagined outcomes via a critic and to improve multi‑step plans through an optimizer trained offline on world‑model roll‑outs. It is the first approach to fully learn plan improvement and can be attached to any pretrained latent world model. In experiments on visual navigation, arm reaching, and robotic manipulation, RP1 outperforms hand‑designed search algorithms, achieving near‑perfect success while using far fewer roll‑outs and running up to 67× faster than the strongest alternative.