← Back to all news
arXiv AI

DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking

The paper introduces DRMOT, a new dataset and framework for Referring Multi-Object Tracking (RMOT) that incorporates RGB-D data. DRSet contains 2,076 referring expressions across 186 RGB-D sequences, pairing synchronized RGB images, depth maps, and language descriptions—including depth-dependent cues—to evaluate spatial‑semantic detection and tracking. The proposed DRTrack framework uses a Geometric‑Aware GRPO fine‑tuned Multimodal Large Language Model to perform precise detection and robust trajectory association from joint language, RGB, and depth inputs, with experiments demonstrating its effectiveness.

Related stories

arXiv Computer Vision
Sep 4

ORMOT: A Dataset and Framework for Omnidirectional Referring Multi-Object Tracking

The paper introduces ORMOT, a new task that extends Referring Multi‑Object Tracking to omnidirectional 360° imagery, ensuring full scene context for language‑guided tracking. It presents ORSet, a dataset of 27 omnidirectional scenes with 848 language descriptions and 3,401 annotated objects, and introduces ORTrack, an LVLM‑driven framework that performs zero‑shot detection and robust cross‑frame association. Experiments on ORSet show that ORTrack achieves state‑of‑the‑art performance, establishing a strong baseline for future research.

By Zihan Zhou, Sijia Chen, Yanqiu Yu, En Yu, Wenbing Tao
agentsroboticsbenchmarks
More like this →
arXiv AI
Sep 28

DepthEvidence: Unifying Metric Depth Prediction and Geometric Reasoning in Multimodal Language Models

DepthEvidence is a 4B multimodal language model that integrates dense metric depth predictions into language generation. It employs a camera‑conditioned decoder to produce full‑resolution depth maps and a dense‑to‑language interface that converts these predictions into object‑aligned geometry tokens. The model is trained with geometric supervision and instruction tuning, and it sets new state‑of‑the‑art results on a Depth‑VQA benchmark and on instance‑level metric depth estimation across nine datasets.

By Jiangning Wei, Yuan Yao, Miaomiao Cui, Mingsheng Li, Humen Zhong, Shuai Bai, Zhibo Yang
llmscomputer-visionfine-tuningmultimodalbenchmarks
More like this →
arXiv Computer Vision
Oct 2

LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation

LangMap introduces a human‑verified benchmark for language‑conditioned goal navigation (LGN) that spans four hierarchical semantic levels—scene, room, region, and instance—within real‑world indoor 3D scans. The dataset, built on HM3D, contains 18K tasks with concise and detailed descriptions for 414 object categories, and its contrastive annotation protocol ensures high‑quality, discriminative region and instance labels. Evaluation shows that LangMap’s descriptions improve text‑to‑view matching accuracy by 23 points over GOAT‑Bench and achieve a 92.5% unique‑and‑correct match rate in an independent human audit, while a proposed RGB‑only baseline, PlaNaVid, attains top‑tier success rates without depth or 3D scene representations.

By Bo Miao, Weijia Liu, Jun Luo, Lachlan Shinnick, Jian Liu, Thomas Hamilton-Smith, Yuhe Yang, Zijie Wu, Vanja Videnovic, Feras Dayoub, Anton van den Hengel
llmsagentsroboticsmultimodalbenchmarks
More like this →
arXiv Computer Vision
Oct 2

GenCOPE: Syn2Real Generalized Category-Level Object Pose Estimation for Robotic Picking

GenCOPE introduces a synthetic-to-real (Syn2Real) approach for category-level object pose estimation (COPE) that eliminates the need for labor-intensive real-world data collection. By learning domain-invariant representations through 2D and 3D semantic consistency constraints and employing an end-to-end pose regression framework with 2D-3D cross consistency, the model achieves robust generalization across synthetic and real domains. The architecture relies solely on global features, resulting in a lightweight and efficient design validated on REAL275, Wild6D, and real-world robotic manipulation scenes.

By Jian Liu, Wei Sun, Zhenqi Dai, Hui Yang, Jian Xiao, Nicu Sebe, Na Zhao
computer-visionroboticsbenchmarks
More like this →