arXiv Computer Vision

GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction

GazeFlow is a new framework for egocentric gaze prediction that models gaze as a joint distribution of temporal positions conditioned on both top‑down task cues and bottom‑up visual saliency. It employs conditional flow matching to iteratively transform Gaussian noise into realistic gaze trajectories, using a velocity field informed by video‑encoded visual features and global task queries. On standard benchmarks, GazeFlow outperforms existing methods on per‑frame metrics and produces trajectories that better reflect human gaze dynamics.

arXiv AI
Nov 6

StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos

StreamGaze is a new benchmark that evaluates how well Multimodal Large Language Models (MLLMs) use human gaze signals for temporal and proactive reasoning in streaming videos. It introduces gaze‑guided tasks covering past, present, and future responses, and employs a QA generation pipeline that aligns egocentric videos with raw gaze trajectories. The benchmark reveals significant gaps between current MLLMs and human performance, highlighting limitations in gaze‑based temporal reasoning, intent modeling, and proactive prediction.

By Daeun Lee, Subhojyoti Mukherjee, Branislav Kveton, Ryan A. Rossi, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Mohit Bansal
arXiv Computer Vision
Nov 5

EgoM2P: Egocentric Multimodal Multitask Pretraining

EgoM2P is a masked‑modeling framework that learns from temporally‑aware multimodal tokens to train a large, general‑purpose model for egocentric 4D understanding. It supports multitasking across diverse egocentric perception and synthesis tasks—such as gaze prediction, egocentric camera tracking, monocular depth estimation, and conditional egocentric video synthesis—while matching or surpassing specialist models at a fraction of the computational cost. The authors plan to fully open‑source EgoM2P to advance egocentric vision research.

By Gen Li, Yutong Chen, Yiqian Wu, Kaifeng Zhao, Marc Pollefeys, Siyu Tang
arXiv Computer Vision
Oct 5

Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal LLMs

Gaze Attention is a new mechanism for multimodal large language models that selectively focuses on relevant visual regions during each generation step, rather than attending to all visual tokens. By grouping tokens into spatial regions and using learnable context tokens to retain global information, it reduces attention computation and visual key‑value entries by up to 90%. Experiments on 13 image and 6 video benchmarks show that Gaze Attention matches or outperforms dense‑attention baselines while using fewer visual resources.

By Junha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo, Dongyoon Han, Sangdoo Yun
Hugging Face Trending Papers
Aug 11

Gaze Target Estimation Anywhere with Concepts

Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primarily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and human pose, in order to identify the subject of gaze analysis.

arXiv Computer Vision
Oct 5

Moving Forward with Video Saliency: A New Dataset and Benchmark where Motion Matters

The paper introduces SalTempto, a new video saliency benchmark featuring 224 highly dynamic clips from HACS‑Segments, each with event context and gaze data from up to 16 subjects. It shows that static baselines perform poorly on SalTempto compared to LEDOV, while fine‑tuned temporal models achieve significant gains, yet still leave a large portion of explainable gaze information unaccounted for. The dataset highlights the need for better temporal modeling in video saliency research.

By Susmit Agrawal, Rebecca Wanner, Juliane Verwiebe, Matthias Tangemann, Matthias Bethge, Matthias K\"ummerer