arXiv Computer Vision September 22, 2026 By Huiqiong Li, Zhiting Mei, Anirudha Majumdar, Jingjing Chen, Yu-Gang Jiang, Bin Zhu

LIBERO-VPro: Benchmarking Closed-Loop Visual Robustness of Robotic Foundation Models

Read the original on arXiv Computer Vision →

LIBERO-VPro is a benchmark designed to assess the closed‑loop visual robustness of robotic foundation models by systematically perturbing visual inputs during task execution. It spans four dimensions—Visual Evidence Degradation, Camera Staleness, Visual Source Consistency, and Task‑Relevant Scene Variation—across 12 challenge categories, 96 settings, and 3,296 task‑condition cases. Evaluations on six models over 196,000 simulated episodes and 200 real‑world rollouts show that high nominal performance can hide significant weaknesses in visual grounding, adaptation, and sensitivity to stale or missing observations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Hugging Face Trending Papers

2d ago

LIBERO-VPro: Benchmarking Closed-Loop Visual Robustness of Robotic Foundation Models

Robotic foundation models achieve impressive performance on standard manipulation benchmarks, yet these evaluations typically assume clean, timely, and consistent visual observations throughout execut...

robotics multimodal benchmarks

arXiv AI

Jun 30

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

arXiv:2606. 28385v1 Announce Type: cross Abstract: Recent advances in robot world models enable synthetic video generation for embodied prediction and planning.

By Minh-Loi Nguyen, Nghiem Tuong Diep, Hung Khang Nguyen, Minh Le, Doanh Le Thien, Hoang H. Tran, Dung D. Le, Vu N. Duong, Daniel Sonntag, An Thai Le, Duy Minh Ho Nguyen, Vien Anh Ngo, Tran Van Nhiem

llms agents diffusion robotics multimodal benchmarks safety

arXiv AI

Jul 3

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

arXiv:2603. 06001v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models enable robots to perform manipulation tasks directly from natural language instructions and are increasingly viewed as a foundation for generalist robotic policies.

By Ninghao Zhang, Bin Zhu, Shijie Zhou, Jingjing Chen

robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 17

Can Vision Foundation Models Navigate? Zero-Shot Real-World Evaluation and Lessons Learned

arXiv:2603. 25937v2 Announce Type: replace-cross Abstract: Visual Navigation Models (VNMs) promise generalizable, robot navigation by learning from large-scale visual demonstrations.

By Maeva Guerrier, Karthik Soma, Jana Pavlasek, Giovanni Beltrame

llms diffusion robotics benchmarks

arXiv AI

Sep 7

RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?

RoboSPA is a large-scale robotic manipulation dataset and benchmark designed to evaluate Vision‑Language‑Action models on fine‑grained spatial reasoning and long‑horizon procedural planning. It contains 10 task categories, 56 base tasks, and 280 variants across five difficulty levels, with 527K trajectories collected from multiple embodiments and scenes. The benchmark introduces diagnostic metrics beyond binary success, revealing that current VLA models struggle with complex spatial relations, precise execution, and memory‑intensive planning.

By Zhenxuan Fan, Bo Zhang, Yutong Lin, Yuqian Yuan, Juekai Lin, Liang Liang, Zhuoyi Huang, Wenqiao Zhang, Juncheng Li, Siliang Tang, Jun Xiao, Yueting Zhuang

agents robotics multimodal benchmarks

arXiv AI

Sep 4

FailBench: How Reliable are VLMs at Judging Robot Task Success?

FailBench is a new benchmark for robot failure detection, containing 2,197 manipulation attempts from 14 public sources, with 75% of failures occurring naturally. The study evaluates 13 vision‑language model (VLM) detectors, finding the best model achieves only 0.77 mean balanced accuracy, and that fine‑tuned failure detectors often underperform general‑purpose VLMs. Performance varies with visual evidence, excelling when object motion is observable but dropping to near chance on contact‑intensive assembly tasks, and input‑level cropping of outcome‑relevant regions improves the top detector by 2.4 percentage points.

By Zaruhi Navasardyan, Tatul Danielyan, Hrant Davtyan

llms robotics fine-tuning multimodal benchmarks safety

One email a morning, machine-written

Related stories

LIBERO-VPro: Benchmarking Closed-Loop Visual Robustness of Robotic Foundation Models

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

Can Vision Foundation Models Navigate? Zero-Shot Real-World Evaluation and Lessons Learned

RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?

FailBench: How Reliable are VLMs at Judging Robot Task Success?