← Back to all news
arXiv AI By Yu Chen, Ruihang Liu, Yangguang Xu, Xinyue Jiang, Mohammed Bennamoun, Farid Boussaid, Xinyuan Qian, Qiuhong Ke

SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding

Read the original on arXiv AI →

SAVU-Bench is a real‑world benchmark designed to evaluate spatial audio‑visual understanding across three capability levels and seven tasks, addressing limitations of prior simulated‑scene datasets. It includes a diagnostic set, SAVU‑Diag, that breaks down reasoning questions into grounding and alignment sub‑tasks, revealing that many errors stem from failures in these prerequisites. A training‑free baseline, SAVU‑EA, improves spatial grounding and joint matching but high‑level reasoning remains difficult, underscoring the need for better audio perception and cross‑modal spatial integration.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jun 10

Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding

arXiv:2606. 10738v1 Announce Type: cross Abstract: Recent multimodal large language models mainly process audio as monaural signals, thereby discarding the spatial cues contained in spatial audio for sound localization, spatial relation reasoning, and spatial scene understanding.

By Zhiyuan Zhu, Yixuan Chen, Yiwen Shao, Wenxiang Guo, Changhao Pan, Yu Zhang, Yuxiang Wang, Wei Liu, Houhua Zhang, Chengkuan Zeng, Wenbo Cheng, Yunxi Liu, Rui Yang, Steve Yves, Liefeng Bo, Zhou Zhao
llmsnlpmultimodal
More like this →
arXiv Machine Learning
6d ago

FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMs

FloorSAV is a framework that enhances audio‑visual large language models (AV‑LLMs) by embedding a dynamic 2D floormap derived from 3D point clouds, camera trajectories, spatial audio cues, and object landmarks. The floormap is fed into the AV‑LLM as a synchronized stream with egocentric video, enabling the model to jointly reason over visual, auditory, and geometric information in a single inference. The authors also introduce SAVED‑Bench, a benchmark for spatial reasoning tasks, and show that FloorSAV improves AV‑LLM performance on both SAVED‑Bench and SAVVY‑Bench, with ground‑truth floormaps highlighting its potential.

By Kyeong-Rae Kim, Sungnyun Kim, Tae-Hyun Oh
llmsagentsroboticsfine-tuningmultimodalbenchmarks
More like this →
arXiv AI
Aug 11

Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

arXiv:2608. 09435v1 Announce Type: new Abstract: Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time.

By Zhi Zeng, Cheng Zhang, Zesheng Yang, Rendong Pi, Jiaying Wu, Di Zhang, Zihan Ma, Guodong Li, Zhou Yang, Yu Xiang, Yifei Zheng, Minnan Luo
llmsreinforcement-learningmultimodalbenchmarks
More like this →