arXiv Computer Vision

Refer-CityFlow and M3Track: A Benchmark Dataset and Reference Framework for Language-Driven Multi-Target Multi-Camera Tracking

The paper introduces Refer-CityFlow, a new benchmark for referring multi-target multi-camera tracking (Refer-MTMCT) that combines natural language queries with cross-camera identity annotations from CityFlow. It covers 96 videos from 40 roadside cameras, 660+ identity-consistent trajectories, 4,362 trajectory-level descriptions, and 7,699 queries, including 1,699 real-user queries. The authors also present M3Track, a modular multimodal framework that uses a large language model to coordinate detection, tracking, and cross-camera association, achieving 44.10 HOTA and 65.40 IDF1 on the real-user query set.

arXiv Computer Vision
Sep 4

ORMOT: A Dataset and Framework for Omnidirectional Referring Multi-Object Tracking

The paper introduces ORMOT, a new task that extends Referring Multi‑Object Tracking to omnidirectional 360° imagery, ensuring full scene context for language‑guided tracking. It presents ORSet, a dataset of 27 omnidirectional scenes with 848 language descriptions and 3,401 annotated objects, and introduces ORTrack, an LVLM‑driven framework that performs zero‑shot detection and robust cross‑frame association. Experiments on ORSet show that ORTrack achieves state‑of‑the‑art performance, establishing a strong baseline for future research.

By Zihan Zhou, Sijia Chen, Yanqiu Yu, En Yu, Wenbing Tao
arXiv Computer Vision
Sep 10

VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models

arXiv:2609.09396v1 Announce Type: new Abstract: As Vision-Language Models (VLMs) advance toward physical deployment, the focus has remained on action-oriented Embodied AI evaluated on subject-centric...

By Zaid Pervaiz Bhat, Nimra Nayyar, Arihant Jain, Lap Fung Chan, John Suchanek, Yu Wang, Varun Praveen, Tomasz Kornuta, Vidya Nariyambut Murali
arXiv Computation and Language
Oct 19

Referring Multi-Object Tracking with Diverse Dynamic Expressions

The paper introduces Referring Multi-Object Tracking (RMOT), a task that uses natural language expressions to guide the prediction of multiple objects over time, addressing limitations in previous work that lacked dynamic language expressiveness. It presents the Refer-KITTI-V2 benchmark, which includes 9,758 semi-automatically generated language prompts for diverse object dynamics. The authors also propose TempRMOT, a Transformer-based framework with a Temporal Enhancement Module that achieves state‑of‑the‑art results on both Refer‑KITTI and Refer‑KITTI‑V2.

By Yani Zhang, Dongming Wu, Wencheng Han, Xingping Dong, Shengcai Liao, Bo Du
arXiv AI
Sep 7

ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search

ARGOS is a new benchmark and agent framework for multi‑camera person search that transforms the task from one‑shot retrieval into interactive reasoning using partial witness clues. The framework requires agents to plan questions, use spatial or temporal tools, and interpret ambiguous natural‑language responses within a limited turn budget, leveraging a Spatio‑Temporal Topology Graph that encodes camera connectivity and transition times. The benchmark includes 2,691 tasks across 14 real‑world scenarios, divided into semantic, spatial, and temporal tracks, and introduces Turn‑Weighted Success (TWS) as a metric that jointly measures correctness and turn efficiency, with current best agents achieving TWS scores of 0.383 and 0.590 on the spatial and temporal tracks respectively.

By Myungchul Kim, Kwanyong Park, Junmo Kim, In So Kweon
arXiv Computer Vision
Sep 3

YesTrack: Referring Multi-Object Tracking via MLLM-based Yes/No Verification

YesTrack introduces a two‑stage referring multi‑object tracking approach that treats the task as a discriminative Yes/No verification problem, directly using multimodal large language models (MLLMs) without generating captions. It adds lightweight temporal consistency constraints—Temporal Confidence Prior (TCP) and Temporal Reference Propagation (TRP)—to improve reliability and efficiency. The method also extends to generic multi‑object tracking as YesTrack‑MOT, achieving state‑of‑the‑art performance on Refer‑KITTI datasets while remaining efficient even with the smallest Qwen3‑VL variant.

By Quansheng Hu, Qin Sun, Qiansen Dai, Jin Ding, Wan Zhang, Xue Zhou, Jianxiao Zou
arXiv Computer Vision
Nov 9

CRTrack-Cap: A New Benchmark for Cross-View Dense Video Object Captioning

CRTrack-Cap is a new benchmark for cross‑view dense video object captioning that links tracklets from multiple camera views into unified global trajectories and generates a single caption per trajectory. The dataset, derived from CRTrack, contains 425 global trajectories, 838 local tracklets across 13 scenes, and canonical captions for 335 identities. A reference framework combining VECA and CEG achieves strong CV‑CHOTA scores, offering a baseline for consistent multi‑camera person description.

By Xiangqun Zhang, Likai Wang, Zekun Qian, Rui Zhou, Ruize Han, Wei Feng
arXiv Computer Vision
Sep 30

ReWorld-Track: A Recursive Event World Model for Language-Guided Multi-Camera Tracking

ReWorld-Track introduces a recursive event world model for language‑guided multi‑camera tracking that explicitly carries association uncertainty into future predictions. By treating candidate matches and waiting as alternative target states, the model updates a persistent recurrent belief that preserves uncertainty across successive observations. This approach improves identity continuity and next‑camera accuracy, achieving HOTA scores of 65.19 on CityFlowV2 and 45.36 on MTMMC, and reducing median arrival‑time error from 0.78 s to 0.71 s.

By Haoyang Wu, Shoudong Han, Chaoyue Li, Sijia Chen, Zhenyang Xie, Wang sihan
arXiv AI
Sep 10

Concord: A Video Relational Algebra for Cross-Modal Query Optimization

Concord introduces a Video Relational Algebra (VRA) that models videos, transcripts, frames, and object tracks, enabling semantic video queries. It applies approximate optimizations to rewrite VRA queries, reducing large language model (MLLM) usage by processing transcripts or using detection and tracking instead of full-video MLLM joins. Experiments on soccer broadcasts and lectures show that Concord sends only a small fraction of video to the MLLM, cutting costs by up to 87%, and improves cross‑camera query accuracy from an F1 of .364 to .813 without any MLLM calls.

By Sultan Muratbek, Charisse Ivana Yeung, Chanwut Kittivorawong, Alvin Cheung
arXiv AI
3d ago

CamChoreo: Benchmarking Temporally Grounded Compositional Camera Motion Understanding for Vision Language Models

CamChoreo introduces a new benchmark for temporally grounded, compositional camera motion understanding, featuring 4,229 single-shot clips with expert temporal annotations across 20 direction-aware labels, many of which involve compound motions. The paper presents a training corpus of ~43K clips and evaluates models using supervised fine‑tuning, a 3D‑injected approach (CamInject), and a distilled method (CamDistill) that embeds geometric knowledge into lightweight camera tokens. CamDistill achieves significant gains over baseline fine‑tuning and demonstrates strong zero‑shot transfer to external benchmarks.

By Dazhao Du, Shiyan Du, Jian Liu, Yongjian Yu, Bohai Gu, Tao Han, Jie Zhang, Hualuo Liu, Eric Liu, Yujia Zhang, Xi Chen, Song Guo
arXiv Machine Learning
Sep 24

Less Language, More Latents: Annotation-Efficient VLAs for Driving

The paper introduces Latent Action Driving Annotations (LADA), a three‑stage pipeline that converts large amounts of unlabelled observation‑trajectory data into a language‑conditioned driving model. First, a latent action model with a vector‑quantised bottleneck learns a compact codebook of vehicle intents. Then, a small set of language‑annotated examples trains a vision‑language translator to map observations and instructions into this codebook, and finally a VLA is trained on observation‑latent‑action pairs across the full corpus. Using less than 5% of language annotations, LADA attains a Driving Score of 87.98 and a Success Rate of 70.46% on Bench2Drive, matching or surpassing fully supervised baselines.

By Alexey Zakharov, Kemal Oksuz, Puneet K. Dokania
arXiv AI
Oct 15

DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking

The paper introduces DRMOT, a new dataset and framework for Referring Multi-Object Tracking (RMOT) that incorporates RGB-D data. DRSet contains 2,076 referring expressions across 186 RGB-D sequences, pairing synchronized RGB images, depth maps, and language descriptions—including depth-dependent cues—to evaluate spatial‑semantic detection and tracking. The proposed DRTrack framework uses a Geometric‑Aware GRPO fine‑tuned Multimodal Large Language Model to perform precise detection and robust trajectory association from joint language, RGB, and depth inputs, with experiments demonstrating its effectiveness.

By Sijia Chen, Shaofeng Liang, Lijuan Ma, Yanqiu Yu, Nan Li, Wenshuo Chen, Runwei Guan, Dongming Wu, En Yu, Yutao Yue