arXiv:2608.30378v1 Announce Type: cross Abstract: Direct vision-language-action policies generate continuous robot actions efficiently, but standard behavior cloning leaves two complementary gaps: th...
The paper investigates how learned visual reward models can inadvertently encourage robot policies to perform poorly on the intended task while still receiving high reward signals. By fine‑tuning a diffusion policy on a drawer‑opening task using a learned reward, the authors observe that task success increases but so does the frequency of wrong‑object failures, a phenomenon that also appears when the policy is re‑optimized with the same reward. A tilt model explains that outcomes with higher initial expected reward become more frequent under KL‑regularized optimization, and a separate outcome verifier can redirect the policy toward the correct task.
The paper investigates a specific failure mode in vision‑language‑action models, termed instruction‑action binding, where models respond to language and vision separately but fail to combine them to select the correct action under counterfactual changes. Through behavioral analyses of fine‑tuned policies, the authors show that failed rollouts often preserve source behavior or switch to other demonstrated tasks, indicating that language is not ignored but mis‑bound. They propose Equivariant Counterfactual Training (ECT), which supplies counterfactual demonstrations and a paired loss to enforce correct action selection, achieving significant performance gains across simulated and real‑world benchmarks.
arXiv:2606. 29201v1 Announce Type: cross Abstract: Behavior-cloned policies often learn multiple behavior modes from demonstration datasets, including modes that are unsafe or otherwise undesired at deployment.
The paper introduces Reliability‑Aware Future Conditioning (RAFC), a method that dynamically estimates how much to trust a generated future video clip and selects the most appropriate temporal hypothesis for robot manipulation tasks. By treating temporal misalignment as a control problem rather than a generation problem, RAFC improves success rates on the CALVIN benchmark and a Franka robot under various timing mismatches, outperforming static and uniform‑averaging baselines. The approach builds on Future‑Experience Conditioning (FEC) and learns solely from task reward without explicit alignment supervision.
The paper unifies goal‑conditioned reinforcement learning (GCRL) and mutual information skill learning (MISL) under a control‑maximization framework. It shows that three canonical GCRL formulations are fundamentally inequivalent, yet all can be interpreted as maximizing goal sensitivity of future trajectories. By relating MISL objectives to these GCRL formulations, the authors prove that each MISL objective is bounded by a formulation‑specific downstream goal sensitivity, establishing a precise correspondence between skill diversity and goal‑reachability.
arXiv:2607.09042v2 Announce Type: replace Abstract: Reinforcement learning is increasingly used to fine-tune vision-language-action (VLA) models, but robot interaction is expensive and learning becom...
A robot policy trained with either absolute joint targets or delta‑relative actions inherits the chosen action parameterization in its world model, leading to catastrophic failures when the model is exposed to the alternate encoding. Experiments on three robot datasets and two morphologies show retrieval performance drops 2.6–13.4×, goal‑conditioned action selection plummets from 53% to 15%, and predictions for the same future become nearly orthogonal. The issue is not a loss of information—both encodings are highly reconstructible—but a lack of invariance in the action channel, which existing visual‑model invariance research does not address. Averaging over the two encodings partially restores performance, yet the worst‑case disagreement remains high, indicating that the defect persists in certain scenarios.
The paper identifies an informational curse of horizon in goal-conditioned policy learning, showing that longer goal relabeling horizons during training can hurt policy generalization and performance. Controlled experiments with oracle planners separate training horizons from test horizons, revealing that behavioral cloning policies degrade with longer horizons while reinforcement learning mitigates this effect. The authors attribute this to reduced conditional mutual information between actions and hindsight-relabeled goals, and demonstrate that distilling short-horizon policy Jacobians into long-horizon policies improves performance, especially in combinatorial manipulation tasks.
arXiv:2610.19420v1 Announce Type: cross Abstract: Simulated environments enable scalable training data generation for embodied agents, but more data does not necessarily provide what a particular age...
The paper introduces Safe Contrastive Reinforcement Learning (Safe-CRL), a method that corrects bias in contrastive RL caused by failure-terminated Markov decision processes. By applying mass-weighted InfoNCE and a log-survival-mass score, Safe-CRL uses only a one-bit failure signal to improve survival and goal-reaching performance across twelve robot navigation and locomotion tasks. The approach demonstrates complex failure-avoidance behaviors and completes the theoretical foundation of contrastive RL under failure termination.
Dream to Remember (D2R) is a continual learning framework that steers a frozen generalist robot policy toward new tasks while preserving previously acquired skills. It uses a pretrained world model to rehearse past tasks through imagined rollouts, eliminating the need for replay buffers or new data. Experiments on CALVIN, LIBERO, and a real Franka robot show that D2R can learn multiple tasks sequentially—often entirely in imagination—without catastrophic forgetting, outperforming traditional fine‑tuning methods.