arXiv Machine Learning
What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Sep 1
PAVE: Predictive Alignment and Value-Guided Evolution for World-Action Policies
arXiv:2608.30378v1 Announce Type: cross Abstract: Direct vision-language-action policies generate continuous robot actions efficiently, but standard behavior cloning leaves two complementary gaps: th...
arXiv AI
Oct 5
CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization
The paper investigates how learned visual reward models can inadvertently encourage robot policies to perform poorly on the intended task while still receiving high reward signals. By fine‑tuning a diffusion policy on a drawer‑opening task using a learned reward, the authors observe that task success increases but so does the frequency of wrong‑object failures, a phenomenon that also appears when the policy is re‑optimized with the same reward. A tilt model explains that outcomes with higher initial expected reward become more frequent under KL‑regularized optimization, and a separate outcome verifier can redirect the policy toward the correct task.
arXiv Machine Learning
Oct 2
When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models
The paper investigates a specific failure mode in vision‑language‑action models, termed instruction‑action binding, where models respond to language and vision separately but fail to combine them to select the correct action under counterfactual changes. Through behavioral analyses of fine‑tuned policies, the authors show that failed rollouts often preserve source behavior or switch to other demonstrated tasks, indicating that language is not ignored but mis‑bound. They propose Equivariant Counterfactual Training (ECT), which supplies counterfactual demonstrations and a paired loss to enforce correct action selection, achieving significant performance gains across simulated and real‑world benchmarks.
arXiv AI
Jun 30
Behavior Uncloning: Distilling Mode Redirection into Policy Weights without Inference-Time Steering
arXiv:2606. 29201v1 Announce Type: cross Abstract: Behavior-cloned policies often learn multiple behavior modes from demonstration datasets, including modes that are unsafe or otherwise undesired at deployment.
arXiv Machine Learning
Oct 9
Reliability-Aware Future Conditioning for Temporally Robust Robot Manipulation
The paper introduces Reliability‑Aware Future Conditioning (RAFC), a method that dynamically estimates how much to trust a generated future video clip and selects the most appropriate temporal hypothesis for robot manipulation tasks. By treating temporal misalignment as a control problem rather than a generation problem, RAFC improves success rates on the CALVIN benchmark and a Franka robot under various timing mismatches, outperforming static and uniform‑averaging baselines. The approach builds on Future‑Experience Conditioning (FEC) and learns solely from task reward without explicit alignment supervision.
arXiv Machine Learning
Oct 23
Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization
The paper unifies goal‑conditioned reinforcement learning (GCRL) and mutual information skill learning (MISL) under a control‑maximization framework. It shows that three canonical GCRL formulations are fundamentally inequivalent, yet all can be interpreted as maximizing goal sensitivity of future trajectories. By relating MISL objectives to these GCRL formulations, the authors prove that each MISL objective is bounded by a formulation‑specific downstream goal sensitivity, establishing a precise correspondence between skill diversity and goal‑reachability.