FloorSAV is a framework that enhances audio‑visual large language models (AV‑LLMs) by embedding a dynamic 2D floormap derived from 3D point clouds, camera trajectories, spatial audio cues, and object landmarks. The floormap is fed into the AV‑LLM as a synchronized stream with egocentric video, enabling the model to jointly reason over visual, auditory, and geometric information in a single inference. The authors also introduce SAVED‑Bench, a benchmark for spatial reasoning tasks, and show that FloorSAV improves AV‑LLM performance on both SAVED‑Bench and SAVVY‑Bench, with ground‑truth floormaps highlighting its potential.
By Kyeong-Rae Kim, Sungnyun Kim, Tae-Hyun Oh
arXiv:2609.39588v1 Announce Type: new
Abstract: We push the frontier of large-scale spatial intelligence in Vision-Language Models (VLMs) and introduce the first benchmark that probes geographical la...
By Aravindh Mahendran, Michael King, Matthew Koichi Grimes, Antoine Yang, Tyler Zhu, Joseph Heyward, Tengda Han, Shiry Ginosar, Chen Sun, Dima Damen, Simon Osindero, Noah Snavely, Simon Lynen, Jo\~ao Carreira, Viorica P\u{a}tr\u{a}ucean
arXiv:2609.23407v2 Announce Type: replace-cross
Abstract: Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challengin...
By Ruixun Liu, Yuxuan Wang, Jiacheng Xie, Yuhuan You, Donghua Cai, Junming Lin, Xiong-Hui Chen, Zhifang Guo, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Yiwu Zhong
arXiv:2609.23407v1 Announce Type: cross
Abstract: Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for em...
By Ruixun Liu, Yuxuan Wang, Jiacheng Xie, Yuhuan You, Donghua Cai, Junming Lin, Xiong-Hui Chen, Zhifang Guo, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Yiwu Zhong
The paper introduces HLR-AVSceneQA, a benchmark that tests audio‑visual large language models on a hear‑localize‑reason hierarchy, assessing recognition of sounds, localization in egocentric and allocentric views, and reasoning about their relation to the visual scene. It also presents HLR‑LLM, a model that adds a binaural spatial audio branch to a strong audio‑visual foundation and is trained with a curriculum including chain‑of‑thought supervision for multi‑step spatial reasoning. Experiments show that existing AV‑LLMs are limited in spatial grounding and relational reasoning, while HLR‑LLM markedly improves spatial audio‑visual scene understanding without hurting semantic perception.
By Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung, Oh Hyun-Bin, Hyeonggon Ryu, David Harwath, Tae-Hyun Oh
arXiv:2608. 09435v1 Announce Type: new Abstract: Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time.
By Zhi Zeng, Cheng Zhang, Zesheng Yang, Rendong Pi, Jiaying Wu, Di Zhang, Zihan Ma, Guodong Li, Zhou Yang, Yu Xiang, Yifei Zheng, Minnan Luo