← Back to all news
arXiv AI By Zongyao Zhao, Cheng-Jie Zhao, Ruihang Jiang, Bing Zheng, Zhaolin Wang, Yuanwei Liu

When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning

Read the original on arXiv AI →

The paper introduces Martingale GRPO (MART‑GRPO), a method for selectively truncating gradient contributions in vision‑language‑action (VLA) reinforcement learning. By defining stopping conditions based on success‑probability and policy‑score martingales, the approach removes gradient terms that would otherwise penalize potentially useful recovery actions while preserving the expectation of the GRPO estimator. Experiments on RoboTwin 2.0 with OpenVLA‑OFT demonstrate that MART‑GRPO yields higher success rates across multiple tasks compared to full‑trajectory GRPO.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Aug 19

Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groups

Prism‑GRPO enhances the GRPO reinforcement‑learning algorithm for vision‑language‑action policies by adding a weighted trajectory‑level execution‑quality score to binary success rewards. This approach splits groups with identical outcomes into a quality spectrum, preserving training signal while ensuring successes always outrank failures. Experiments on four RoboTwin tasks show Prism‑GRPO achieves higher success and quality at matched rollout budgets, reaching target success rates with up to 56% fewer rollouts and mitigating reward‑hacking behaviors that transfer to real‑robot deployment.

By Zeyun Deng, Yuzhe Lu, Yawei Wang, Linbo Liu, Qing Ping, Han Ding, Guande Wu, Panpan Xu, Jun Huan
reinforcement-learningroboticsmultimodalsafety
More like this →
arXiv AI
4d ago

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models

RePO‑VLA is a recovery‑driven policy optimization framework for Vision‑Language‑Action models that separates success, recovery, and failure trajectories. It introduces Recovery‑Aware Initialization, a Progress‑Aware Semantic Value Function, and Value‑Conditioned Refinement to learn from both successful and failed rollouts, enabling the policy to prefer high‑progress actions without online failure detectors. The authors also present the Failure‑Recovery Benchmark (FRBench) and demonstrate that RePO‑VLA raises adversarial success rates from 20% to 75% on average in simulated and real‑world bimanual tasks.

By Weijia Liufu, Xiaoyu Guo, Ruiyi Chen, Jingzhi Liu, Kaidong Zhang, Xiwen Liang, Jianqi Lin, Dawei Sun, Yuze Wang, Rongtao Xu, Bingqian Lin, Bowen Yang, Tongtong Cao, Bowen Peng, Dongyu Zhang, Guangrun Wang, Min Wang, Liang Lin, Xiaodan Liang
roboticsmultimodalbenchmarkssafety
More like this →
arXiv Machine Learning
Oct 2

When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models

The paper investigates a specific failure mode in vision‑language‑action models, termed instruction‑action binding, where models respond to language and vision separately but fail to combine them to select the correct action under counterfactual changes. Through behavioral analyses of fine‑tuned policies, the authors show that failed rollouts often preserve source behavior or switch to other demonstrated tasks, indicating that language is not ignored but mis‑bound. They propose Equivariant Counterfactual Training (ECT), which supplies counterfactual demonstrations and a paired loss to enforce correct action selection, achieving significant performance gains across simulated and real‑world benchmarks.

By Hung-Jen Chen, Yu-Hsun Hou, Yan-Hong Chen, Yan-Fu Chen, Binghua Cai, Min Sun, Chun-Yi Lee
fine-tuningmultimodal
More like this →