arXiv Computer Vision

ORMOT: A Dataset and Framework for Omnidirectional Referring Multi-Object Tracking

The paper introduces ORMOT, a new task that extends Referring Multi‑Object Tracking to omnidirectional 360° imagery, ensuring full scene context for language‑guided tracking. It presents ORSet, a dataset of 27 omnidirectional scenes with 848 language descriptions and 3,401 annotated objects, and introduces ORTrack, an LVLM‑driven framework that performs zero‑shot detection and robust cross‑frame association. Experiments on ORSet show that ORTrack achieves state‑of‑the‑art performance, establishing a strong baseline for future research.

arXiv AI

Oct 15

DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking

The paper introduces DRMOT, a new dataset and framework for Referring Multi-Object Tracking (RMOT) that incorporates RGB-D data. DRSet contains 2,076 referring expressions across 186 RGB-D sequences, pairing synchronized RGB images, depth maps, and language descriptions—including depth-dependent cues—to evaluate spatial‑semantic detection and tracking. The proposed DRTrack framework uses a Geometric‑Aware GRPO fine‑tuned Multimodal Large Language Model to perform precise detection and robust trajectory association from joint language, RGB, and depth inputs, with experiments demonstrating its effectiveness.

By Sijia Chen, Shaofeng Liang, Lijuan Ma, Yanqiu Yu, Nan Li, Wenshuo Chen, Runwei Guan, Dongming Wu, En Yu, Yutao Yue

llms agents robotics fine-tuning multimodal

arXiv Computer Vision

Sep 10

VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models

arXiv:2609.09396v1 Announce Type: new Abstract: As Vision-Language Models (VLMs) advance toward physical deployment, the focus has remained on action-oriented Embodied AI evaluated on subject-centric...

By Zaid Pervaiz Bhat, Nimra Nayyar, Arihant Jain, Lap Fung Chan, John Suchanek, Yu Wang, Varun Praveen, Tomasz Kornuta, Vidya Nariyambut Murali

llms nlp robotics multimodal benchmarks safety

arXiv Computer Vision

Aug 25

Motion-Aware Vision-Reference Alignment for Referring Multi-Object Tracking

arXiv:2511.17681v2 Announce Type: replace Abstract: Referring Multi-Object Tracking (RMOT) extends conventional multi-object tracking (MOT) by introducing natural language references for multi-modal...

By Weiyi Lv, Ning Zhang, Hanyang Sun, Haoran Jiang, Kai Zhao, Yixiao Gu, Jing Xiao, Dan Zeng

llms multimodal benchmarks safety

arXiv Computer Vision

Sep 3

YesTrack: Referring Multi-Object Tracking via MLLM-based Yes/No Verification

YesTrack introduces a two‑stage referring multi‑object tracking approach that treats the task as a discriminative Yes/No verification problem, directly using multimodal large language models (MLLMs) without generating captions. It adds lightweight temporal consistency constraints—Temporal Confidence Prior (TCP) and Temporal Reference Propagation (TRP)—to improve reliability and efficiency. The method also extends to generic multi‑object tracking as YesTrack‑MOT, achieving state‑of‑the‑art performance on Refer‑KITTI datasets while remaining efficient even with the smallest Qwen3‑VL variant.

By Quansheng Hu, Qin Sun, Qiansen Dai, Jin Ding, Wan Zhang, Xue Zhou, Jianxiao Zou

llms multimodal benchmarks safety

arXiv Computation and Language

Oct 15

VLX-Go: Robust Embodied Tracking through Multi-Stage Learning and On-Policy Refinement

arXiv:2610.19361v1 Announce Type: cross Abstract: Embodied visual tracking requires a robot to identify a designated target from visual observations and control its own motion to follow that target t...

By Kyusong Lee, Heting Ying, Tiancheng Zhao

llms robotics multimodal safety

arXiv Computation and Language

Oct 19

Referring Multi-Object Tracking with Diverse Dynamic Expressions

The paper introduces Referring Multi-Object Tracking (RMOT), a task that uses natural language expressions to guide the prediction of multiple objects over time, addressing limitations in previous work that lacked dynamic language expressiveness. It presents the Refer-KITTI-V2 benchmark, which includes 9,758 semi-automatically generated language prompts for diverse object dynamics. The authors also propose TempRMOT, a Transformer-based framework with a Temporal Enhancement Module that achieves state‑of‑the‑art results on both Refer‑KITTI and Refer‑KITTI‑V2.

By Yani Zhang, Dongming Wu, Wencheng Han, Xingping Dong, Shengcai Liao, Bo Du

llms benchmarks

arXiv AI

Jul 7

Generative Semantic Multi-Object Tracking: A Large-Scale Benchmark and an MLLM-Driven Reasoning Framework

arXiv:2601. 06550v3 Announce Type: replace-cross Abstract: Semantic Multi-Object Tracking (SMOT) is evolving from purely geometric localization toward comprehensive video understanding.

By Pan Liao, Feng Yang, Di Wu, Jinwen Yu, Wang Zhao, Dingwen Zhang

llms multimodal benchmarks safety

arXiv Machine Learning

Jul 3

COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm

arXiv:2603. 24016v2 Announce Type: replace-cross Abstract: Multi-Object Tracking (MOT) has traditionally focused on a few specific categories, restricting its applicability to real-world scenarios involving diverse objects.

By Zekun Qian, Wei Feng, Ruize Han, Junhui Hou

benchmarks

arXiv AI

Jul 16

ScanFocus: A Coarse-to-Fine Framework for Spatio-Temporal Video Grounding

arXiv:2607. 13421v1 Announce Type: cross Abstract: Spatio-Temporal Video Grounding (STVG) aims to retrieve the visual trajectory of a specific object from a video stream as described by a natural language expression.

By Kai Chen, Ming Dai, Wenxuan Cheng, Wankou Yang

multimodal benchmarks

arXiv Computer Vision

Sep 1

Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking

arXiv:2608.29126v1 Announce Type: new Abstract: Referring single-object tracking enables language-grounded target initialization and subsequent tracking by jointly leveraging semantic cues and visual...

By Han Wang, Yuxuan Liu, Yuhan Sun, Jian Yang, Xiaotong Xu, Yixuan Lv, Zhuang Zhou, Shengyang Li

llms multimodal benchmarks safety

arXiv AI

Jun 2

CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objects

arXiv:2510. 14904v4 Announce Type: replace-cross Abstract: Dense Video Object Captioning (DVOC) is the task of jointly detecting, tracking, and captioning object trajectories in a video, requiring the ability to understand spatio-temporal details and describe them in natural language.

By Gabriel Fiastre, Antoine Yang, Cordelia Schmid

computer-vision multimodal benchmarks

arXiv Computer Vision

3d ago

PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?

arXiv:2509.02807v2 Announce Type: replace Abstract: Multi-modal large language models (MLLMs) have shown impressive generalization across tasks using images and text modalities. While their extension...

By Mennatullah Siam

llms nlp multimodal benchmarks

Related stories