What Egocentric Video Embeddings Encode and How to Combine Them with Inertial Sensors for Human Activity Recognition

The paper investigates what information is captured by egocentric video embeddings from models such as VideoMAEv2, CLIP, DINOv2, and VC-1, finding that the leading principal components largely reflect where the head‑mounted camera is looking. It also explores how to combine these video embeddings with body‑worn inertial sensor data for human activity recognition, showing that propagating inertial predictions over a video‑based graph can significantly improve classification performance (macro‑F1 from 0.648 to 0.852). The study demonstrates that video and inertial modalities play distinct roles—video identifies similar clips within a set, while inertial data labels the activities—yet fusing them directly in a single classifier or adding inertial edges to the graph does not yield better results.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

• rag