arXiv Computer Vision By Sheng Zhao, Weikai Lin, Yuhao Zhu

GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction

Read the original on arXiv Computer Vision →

GazeFlow is a new framework for egocentric gaze prediction that models gaze as a joint distribution of temporal positions conditioned on both top‑down task cues and bottom‑up visual saliency. It employs conditional flow matching to iteratively transform Gaussian noise into realistic gaze trajectories, using a velocity field informed by video‑encoded visual features and global task queries. On standard benchmarks, GazeFlow outperforms existing methods on per‑frame metrics and produces trajectories that better reflect human gaze dynamics.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

arXiv AI
Nov 6

StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos

StreamGaze is a new benchmark that evaluates how well Multimodal Large Language Models (MLLMs) use human gaze signals for temporal and proactive reasoning in streaming videos. It introduces gaze‑guided tasks covering past, present, and future responses, and employs a QA generation pipeline that aligns egocentric videos with raw gaze trajectories. The benchmark reveals significant gaps between current MLLMs and human performance, highlighting limitations in gaze‑based temporal reasoning, intent modeling, and proactive prediction.

By Daeun Lee, Subhojyoti Mukherjee, Branislav Kveton, Ryan A. Rossi, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Mohit Bansal
arXiv Computer Vision
Nov 5

EgoM2P: Egocentric Multimodal Multitask Pretraining

EgoM2P is a masked‑modeling framework that learns from temporally‑aware multimodal tokens to train a large, general‑purpose model for egocentric 4D understanding. It supports multitasking across diverse egocentric perception and synthesis tasks—such as gaze prediction, egocentric camera tracking, monocular depth estimation, and conditional egocentric video synthesis—while matching or surpassing specialist models at a fraction of the computational cost. The authors plan to fully open‑source EgoM2P to advance egocentric vision research.

By Gen Li, Yutong Chen, Yiqian Wu, Kaifeng Zhao, Marc Pollefeys, Siyu Tang
arXiv Computer Vision
Oct 5

Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal LLMs

Gaze Attention is a new mechanism for multimodal large language models that selectively focuses on relevant visual regions during each generation step, rather than attending to all visual tokens. By grouping tokens into spatial regions and using learnable context tokens to retain global information, it reduces attention computation and visual key‑value entries by up to 90%. Experiments on 13 image and 6 video benchmarks show that Gaze Attention matches or outperforms dense‑attention baselines while using fewer visual resources.

By Junha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo, Dongyoon Han, Sangdoo Yun