arXiv AI
Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Oct 12
Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target
The paper introduces Cross-Behavior Policy Optimization (CBPO), a method that unifies offline demonstrations and policy-generated rollouts into a single regression loss using a fixed-reference target. CBPO eliminates the need for separate supervised fine‑tuning and reinforcement learning objectives, allowing positive and negative offline completions, reference responses, and current and historical rollouts to share a common reward‑labeled format and replay buffer. Experiments on two model scales and four mathematical reasoning benchmarks show that CBPO, especially the CBPO‑Unified variant, outperforms baseline methods in avg@32 and pass@32 metrics, with historical rollouts providing additional gains.
arXiv Machine Learning
Aug 28
Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals
The paper introduces Safe Contrastive Reinforcement Learning (Safe-CRL), a method that corrects bias in contrastive RL caused by failure-terminated Markov decision processes. By applying mass-weighted InfoNCE and a log-survival-mass score, Safe-CRL uses only a one-bit failure signal to improve survival and goal-reaching performance across twelve robot navigation and locomotion tasks. The approach demonstrates complex failure-avoidance behaviors and completes the theoretical foundation of contrastive RL under failure termination.
arXiv Machine Learning
Oct 8
An Informational Curse of Horizon in Goal-Conditioned Policy Learning
The paper identifies an informational curse of horizon in goal-conditioned policy learning, showing that longer goal relabeling horizons during training can hurt policy generalization and performance. Controlled experiments with oracle planners separate training horizons from test horizons, revealing that behavioral cloning policies degrade with longer horizons while reinforcement learning mitigates this effect. The authors attribute this to reduced conditional mutual information between actions and hindsight-relabeled goals, and demonstrate that distilling short-horizon policy Jacobians into long-horizon policies improves performance, especially in combinatorial manipulation tasks.
arXiv Machine Learning
Jun 25
Consistent Zero-Shot Imitation with Contrastive Goal Inference
arXiv:2510. 17059v2 Announce Type: replace Abstract: Zero-shot imitation learning requires an agent to reproduce expert behavior from a single demonstration without additional environment interaction or gradient updates at test time.
arXiv Machine Learning
Sep 29
From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning
The paper introduces Q-Target Pretrained Transformers (QTPT), a method that replaces supervised behavior cloning with a Bellman-style Q‑target objective for in‑context reinforcement learning. QTPT retains the context‑conditioned Transformer architecture but learns to estimate action values using rewards and transitions from the context, rather than merely imitating offline actions. The authors provide theoretical analysis in stochastic linear bandits and finite‑horizon MDPs, demonstrating improved robustness to weak or suboptimal data, and empirically show gains over supervised pretraining on controlled RL benchmarks and extensions to D4RL Kitchen and AntMaze.
arXiv AI
Sep 30
Counterfactual Rollout Replay: Forkable Environments as Free Process Rewards for Software Engineering Agents
arXiv:2609.33875v2 Announce Type: replace-cross Abstract: Outcome-only reinforcement learning gives software engineering (SWE) agents a terminal success signal but little direct guidance about interm...