← Back to all news
arXiv Computer Vision

Fine-grained Human Motion Understanding with Language Models

Fine-grained Human Motion Understanding with Language Models introduces FiGMo, an LLM-based model that represents motion as timestamp‑grounded skeletal poses, enabling reasoning over action order and duration. The model uses a unified pose encoder for both 2D and 3D skeletons, optionally incorporating RGB context, and relies on newly constructed pose‑to‑motion supervision to address limited pose‑level grounding. Experiments demonstrate that timestamps improve temporal localization and preserve physical timing under subsampling, leading FiGMo to achieve state‑of‑the‑art results on motion question answering, dense motion captioning, and action recognition with detected poses.

Related stories

arXiv Computer Vision
Aug 27

Pose-Anchored Optical Flow for Low-Latency Human Action Anticipation in Human-Robot Teaming

Pose-Anchored Optical Flow for Low-Latency Human Action Anticipation in Human-Robot Teaming proposes PoseOFF, a representation that captures local motion around human joints by conditioning optical flow extraction on pose. This structured motion representation aligns with human kinematics and improves early action recognition accuracy across multiple datasets and backbones. PoseOFF achieves comparable or better performance while observing less of the action sequence, making it suitable for real‑time, resource‑constrained robotic systems.

By Lewis de Zoete Grundy, Chris McCarthy, Christopher Fluke
computer-visionroboticsbenchmarks
More like this →
arXiv Computer Vision
Aug 26

SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling

arXiv:2608.24334v1 Announce Type: new Abstract: Discrete motion representations have substantially advanced autoregressive text-to-motion generation. However, most motion tokenizers are optimized for...

By Tianlv Huang, Hetian Guo, Ziyi Cai, Song Wang, Yanping Zhang, Zipei Fan, Xuan Song, Guangming Wu, Xin Zheng
More like this →
arXiv Computer Vision
Oct 12

ELMA: Benchmarking Anaphoric Compositions for Long-Term Text-to-Motion Generation

The paper introduces ELMA, a new episodic long‑term motion dataset designed to benchmark anaphoric compositions in text‑to‑motion generation. ELMA contains densely annotated 30‑second clips extracted from in‑the‑wild videos, preserving cross‑segment dependencies that current models struggle to handle. Fine‑tuning state‑of‑the‑art models on ELMA improves both global physical realism and semantic alignment, establishing a robust baseline for long‑term motion synthesis.

By Shun Kato, Dixin Yang, Kotaro Amaya, Yuiga Wada, Kazumi Fukuda, Takashi Shibuya, Yuki Mitsufuji, Mariko Isogawa
fine-tuningbenchmarkssafety
More like this →
arXiv Computer Vision
Sep 16

GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos

GeoLAM is a framework that learns geometry‑grounded latent actions from unlabeled human videos. It uses future‑frame reconstruction with a frozen geometric feature hierarchy and motion supervision from a 4D geometry teacher to capture 3D displacement, image‑plane motion, and surface‑orientation changes. After pretraining, the representation serves as transition targets for a world‑action model trained on robot demonstrations, enabling denoised latent actions and executable action chunks without requiring hand‑pose annotations or future‑video generation during deployment.

By Yifan Xie, Hekun Tian, Jinkun Liu, YuAn Wang, Qiao Sun, Wenbo Ding
diffusionroboticsbenchmarks
More like this →