arXiv Computation and Language

Free the Language Model From the Vision Encoder: Semantic Serialization as a Perception Interface for Small Language Models

The paper introduces a perception interface that separates vision from language in vision‑language models. A frozen perception stack detects objects, a deterministic semantic serializer converts the perceived state into text, and a standard text‑only large language model (LLM) answers questions. Experiments on a campus‑robot benchmark show that this serialized interface outperforms a zero‑shot VLM of the same language‑model size, especially as the language model shrinks, and that the advantage persists under paraphrase and different supervision regimes.

arXiv Machine Learning
Jun 18

Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

arXiv:2606. 19297v1 Announce Type: new Abstract: Embodied Vision-Language-Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation.

By Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin, Nikita Kurlaev, Daria Pugacheva, Albina Burlova, Mikhail Kolosov, Denis Shepelev, Andrey Kuznetsov, Elena Tutubalina, Aleksandr I. Panov, Alexey K. Kovalev, Vlad Shakhuro
Hugging Face Trending Papers
Jun 17

Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

Embodied Vision-Language-Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation. Failures on knowledge-sensitive tasks are ambiguous, conflating missing knowledge with poor generalization of low-level control.

arXiv Machine Learning
Nov 2

Latent-Lens: Visual Perception in Small Language Models Through Latent Communication

Latent‑Lens is a lightweight codec that enables a frozen vision‑language model (VLM) to communicate visual information to a frozen small language model (SLM) directly in latent space, avoiding the lossy bottleneck of discrete text. The codec uses a domain encoder to project VLM hidden states into the SLM’s embedding space and an attention decoder (a LoRA) that teaches the SLM to attend to these visual prefixes, training only about 1.6 M parameters (0.2 % of the combined model size). On visual question answering tasks (CLEVR, GQA) and image captioning (Flickr8k), Latent‑Lens significantly outperforms text‑based communication with the same LoRA budget, achieving up to +69.8 pp improvement on VQA and +18.8 pp on captioning.

By Mo Shahdloo, Faegheh Sardari, Mathew Salvaris
arXiv AI
1d ago

V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations

The paper introduces V-JEPA-LM, a video‑language model that leverages the dynamic representations learned by V‑JEPA video encoders. By applying counterfactual supervision—pixel transformations that create answer‑balanced video variants—the model learns to read motion dynamics such as speed, direction, and order, rather than merely retrieving static attributes. Trained on 4.53 M public samples with 32 visual tokens per frame, V‑JEPA‑LM outperforms existing open‑source video LLMs across eleven benchmarks, with a 4 B model achieving a 72.1 average score.

By Keuntae Kim, Dongjun Kim, Yong Suk Choi
Hugging Face Trending Papers
Jul 29

TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM

Vision-language-action (VLA) models commonly adopt an LLM-centric $V \to L \to A$ pathway, where visual observations are projected into the representation space of a large language model before being decoded into robot actions. Although effective, this design incurs substantial computation and memory overhead at every policy invocation.

arXiv AI
Jul 15

Visual Access Boundaries in Vision-Language Model Reasoning

arXiv:2607. 12815v1 Announce Type: new Abstract: Chain-of-Thought (CoT) prompting is widely used as a test-time scaling strategy for Vision-Language Models (VLMs), but it remains unclear what is extended when VLMs generate longer reasoning traces.

By Hiroto Osaka, Shohei Taniguchi, Gouki Minegishi, Kai Yamashita, Masahiro Suzuki, Yutaka Matsuo
arXiv Computer Vision
Sep 14

AnchorVLN: Geometry-Anchored Vision-Language Grounding Reasoning for Open-Vocabulary Navigation

AnchorVLN is an open‑vocabulary vision‑language navigation system that separates semantic proposals from geometric metrics. It uses a VLM to generate semantics while a geometry module supplies reliable metric quantities such as range and bearing, all within a Model Context Protocol server. The system achieves 64.4% on instruction following and improves object‑reference accuracy, reducing median center error from 3.37 m to 2.48 m.

By Long Giang Vu, Chengkai Yao, Yuxin Liu, FNU Aryan, Rajath Chandrashekar Aralikatti
arXiv AI
Aug 13

G0.5: One Autoregressive Stream for Robot Reasoning and Action

arXiv:2608. 11739v1 Announce Type: cross Abstract: The prevailing recipe for Vision-Language-Action (VLA) models couples a pretrained VLM with a separately trained flow-matching action expert.

By Yicheng Liu, Zibin Dong, Baijun Ye, Tianyuan Yuan, Tao Jiang, Anqi Yang, Shicheng Cao, Haonan Liu, Yue Sun, Zihan Guo, Xiao Liu, Dong Ke, Changxun Pan, Chenru Wu, Tailai Cheng, Xiaoshu Ren, Xinlei Zhang, Jianning Cui, Zijie Zhao, Haoyu Zhang, Kaiming Xu, Haodong Yang, Bowen Zhang, Jiahui Niu, Shaoting Zhu, Shiduo Zhang, Hang Zhao