arXiv:2606. 10738v1 Announce Type: cross Abstract: Recent multimodal large language models mainly process audio as monaural signals, thereby discarding the spatial cues contained in spatial audio for sound localization, spatial relation reasoning, and spatial scene understanding.
arXiv:2608. 09435v1 Announce Type: new Abstract: Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time.
arXiv:2609.23407v2 Announce Type: replace-cross Abstract: Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challengin...
arXiv:2402. 01591v4 Announce Type: replace-cross Abstract: Spatial sound reasoning is a fundamental human skill, enabling us to navigate and interpret our surroundings based on sound.
arXiv:2609.23407v1 Announce Type: cross Abstract: Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for em...
Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time. Yet existing audio-language models often represent clips as global acoustic events, while vision-language models lack the spatial audio cues needed to localize and track individual sources.
The paper introduces HLR-AVSceneQA, a benchmark that tests audio‑visual large language models on a hear‑localize‑reason hierarchy, assessing recognition of sounds, localization in egocentric and allocentric views, and reasoning about their relation to the visual scene. It also presents HLR‑LLM, a model that adds a binaural spatial audio branch to a strong audio‑visual foundation and is trained with a curriculum including chain‑of‑thought supervision for multi‑step spatial reasoning. Experiments show that existing AV‑LLMs are limited in spatial grounding and relational reasoning, while HLR‑LLM markedly improves spatial audio‑visual scene understanding without hurting semantic perception.
arXiv:2606. 14141v1 Announce Type: cross Abstract: Sound events are entities with semantic identities, locations, and trajectories, but current audio-language models usually reason about clips as global event content.
arXiv:2609.38444v1 Announce Type: new Abstract: Text- and image-conditioned world generators can create visually rich 3D environments, yet these worlds often remain silent or rely on soundtracks synt...
FloorSAV is a framework that enhances audio‑visual large language models (AV‑LLMs) by embedding a dynamic 2D floormap derived from 3D point clouds, camera trajectories, spatial audio cues, and object landmarks. The floormap is fed into the AV‑LLM as a synchronized stream with egocentric video, enabling the model to jointly reason over visual, auditory, and geometric information in a single inference. The authors also introduce SAVED‑Bench, a benchmark for spatial reasoning tasks, and show that FloorSAV improves AV‑LLM performance on both SAVED‑Bench and SAVVY‑Bench, with ground‑truth floormaps highlighting its potential.
Sonic-AMG is a novel framework that generates human motion in real time from spatial audio, the first to do so in a streaming manner. It encodes acoustic content and body-relative source geometry as separate tokens, then uses a causal transformer with a flow head to produce continuous motion latents that a causal decoder turns into motion segments. The system introduces rehearsal training and spatial feedback to mitigate training‑inference mismatches, achieving state‑of‑the‑art FID and R‑precision on the SAM benchmark while enabling coherent, real‑time reactions to sound events and source locations.
SAVU-Bench is a real‑world benchmark designed to evaluate spatial audio‑visual understanding across three capability levels and seven tasks, addressing limitations of prior simulated‑scene datasets. It includes a diagnostic set, SAVU‑Diag, that breaks down reasoning questions into grounding and alignment sub‑tasks, revealing that many errors stem from failures in these prerequisites. A training‑free baseline, SAVU‑EA, improves spatial grounding and joint matching but high‑level reasoning remains difficult, underscoring the need for better audio perception and cross‑modal spatial integration.