← Back to all news
arXiv AI By Ruben Hammele, Samira Huber, Mohammad Nazeri, Amirreza Payandeh, Alexandyr T. Card, Aaron Nguyen, Anuj Pokhrel, Daeun Song, Xuesu Xiao, S\"oren Pirk

What Actually Matters? A Paired Evaluation of Inference Settings for $\pi_{0.5}$ on BEHAVIOR-1K

Read the original on arXiv AI →

The paper evaluates inference and data‑collection strategies for Vision‑Language‑Action (VLA) models on the BEHAVIOR‑1K benchmark. Using a fine‑tuned π_{0.5} model that achieved a Q‑score of 0.39 on the public test set, the authors systematically compare previously proposed strategies across all 100 tasks. The results reveal that these strategies do not transfer uniformly to new environments and tasks, highlighting the need for more robust approaches.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

Hugging Face Trending Papers
Jun 17

Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

Embodied Vision-Language-Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation. Failures on knowledge-sensitive tasks are ambiguous, conflating missing knowledge with poor generalization of low-level control.

agentsroboticsfine-tuningmultimodalbenchmarks
More like this →
arXiv Machine Learning
Jun 18

Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

arXiv:2606. 19297v1 Announce Type: new Abstract: Embodied Vision-Language-Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation.

By Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin, Nikita Kurlaev, Daria Pugacheva, Albina Burlova, Mikhail Kolosov, Denis Shepelev, Andrey Kuznetsov, Elena Tutubalina, Aleksandr I. Panov, Alexey K. Kovalev, Vlad Shakhuro
agentsroboticsfine-tuningmultimodalbenchmarks
More like this →
arXiv AI
Sep 18

StarVLA-$\alpha$: Reducing Complexity in Vision-Language-Action Systems

StarVLA-α is a streamlined Vision‑Language‑Action (VLA) model that reduces architectural and pipeline complexity to facilitate systematic study of VLA design choices. By employing a strong VLM backbone and minimal design, it achieves competitive performance across multiple benchmarks (LIBERO, SimplerEnv, RoboTwin, RoboCasa) and outperforms the baseline π₀.₅ by 20% on the RoboChallenge benchmark. The authors plan to release the code to support future VLA research.

By Jinhui Ye, Ning Gao, Senqiao Yang, Jinliang Zheng, Zixuan Wang, Yuxin Chen, Pengguang Chen, Yilun Chen, Shu Liu, Jiaya Jia
agentsroboticsmultimodalbenchmarks
More like this →