← Back to all news
arXiv AI By Christian A. Rogers, Qian Lin, Daniel S. Brown

Do VLAs Know They Will Fail? Interpretable Success and Failure Patterns in Physical AI Systems

Read the original on arXiv AI →

The paper investigates Vision‑Language‑Action (VLA) models used as robot policies, showing that sparse autoencoder (SAE) representations of OpenVLA activations can predict task failure before an unrecoverable state is reached. It identifies specific SAE features whose strong activations correlate with stages of success or failure, and maps these to recurring behavioral patterns by locating temporal intervals of significant activation change. A VLM‑based interpretation pipeline is then developed to generate behavioral descriptions of these patterns, with validation on held‑out intervals.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 28

Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring

The paper introduces Hide-and-Seek, a framework for detecting failures in Vision‑Language‑Action (VLA) models during robot execution. It treats failure detection as a coarsely supervised learning problem, using inter‑trajectory and intra‑trajectory contrastive objectives to localize failure‑indicative actions without step‑level annotations. Experiments on LIBERO, VLABench, and a real‑world robotic platform show that Hide‑and‑Seek achieves state‑of‑the‑art multi‑task failure detection performance across several VLA policies.

By Seongheon Park, Wendi Li, Changdae Oh, Samuel Yeh, Zsolt Kira, Michael Hagenow, Sharon Li
roboticsmultimodalbenchmarks
More like this →
arXiv AI
1d ago

Failure Prediction and Classification in Vision-Language-Action Models Via Calibrated Latent Probing

The paper introduces VLA‑Tug, a lightweight probe that monitors latent layers of Vision‑Language‑Action models to detect failures early and predict failure modes. By supervising on pre‑failure segments, it improves early detection by up to 30% at matched false‑positive rates and maintains valid missed‑detection and false‑alarm bounds in 86% of runs under unseen task shifts, outperforming existing baselines. The method also raises top‑3 accuracy on unseen tasks by up to 44% and supports real‑time, edge‑efficient monitoring for safer deployment.

By Michal P. Podolinsky, Yunhao Yang, Daniel Little, Hong Lu, Rohan Siva, Olin Goudey, Neel P. Bhatt, Matthias Scheutz, Ufuk Topcu
roboticsmultimodal
More like this →
arXiv Machine Learning
Oct 12

VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models

VLA-FAIL introduces a lightweight framework for detecting failures in vision-language-action models used in robotic manipulation. It combines two novel detectors: the first-layer Mahalanobis distance (LLMD) identifies out-of-distribution states by comparing token-wise deviations in last-layer features to training data, while the second, action chunk consistency (ACC), detects failures by spotting inconsistencies between consecutive action chunks in receding-horizon control. The authors also propose a threshold‑independent metric, “AUCPDT,” to evaluate detection accuracy and latency, and show through real‑world and simulation experiments that LLMD and ACC together provide reliable and early failure detection, outperforming more expensive baseline methods.

By Florian Seligmann, Emiliyan Gospodinov, Enes Ulas Dincer, Gerhard Neumann
roboticsmultimodalbenchmarks
More like this →