arXiv:2609.23407v1 Announce Type: cross Abstract: Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for em...
arXiv:2609.23407v2 Announce Type: replace-cross Abstract: Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challengin...
arXiv:2608. 16285v1 Announce Type: cross Abstract: Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues.
FloorSAV is a framework that enhances audio‑visual large language models (AV‑LLMs) by embedding a dynamic 2D floormap derived from 3D point clouds, camera trajectories, spatial audio cues, and object landmarks. The floormap is fed into the AV‑LLM as a synchronized stream with egocentric video, enabling the model to jointly reason over visual, auditory, and geometric information in a single inference. The authors also introduce SAVED‑Bench, a benchmark for spatial reasoning tasks, and show that FloorSAV improves AV‑LLM performance on both SAVED‑Bench and SAVVY‑Bench, with ground‑truth floormaps highlighting its potential.
BinauralVAE is an open‑source pipeline that reconstructs spatial audio using various Variational Autoencoder architectures, including complex‑valued variants, to learn latent representations of binaural signals. The project builds on realistic acoustic data from a simulated robot navigating an environment, providing a foundation for audio‑centric world models. It aims to map the causal link between navigational actions and their acoustic outcomes, positioning sound as a complementary modality for spatial awareness.
arXiv:2607. 15265v1 Announce Type: cross Abstract: We present SceneBind, an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio and language.
The paper introduces CTAN, a Cycle-Temporal Attention Network for audio‑visual embodied navigation. It proposes an Audio‑Visual Reconstruction Cross‑Attention module that uses bidirectional cycle‑consistency to strengthen spatial semantics across visual and acoustic modalities, and a Temporal Cross‑Modal Memory to fuse real‑time multimodal features with historical context. Experiments on Replica and Matterport3D show that CTAN outperforms prior methods in success rate, SPL, and scene navigation accuracy.
arXiv:2610.15956v1 Announce Type: new Abstract: Vision-language-action models for embodied navigation typically rely on RGB observations, limiting their applicability in lights-out industrial environ...
The paper introduces ACF-Net, an optical flow‑guided framework for asymmetric audio‑visual fine‑grained visual categorization (FGVC), addressing challenges where video and audio are not strictly synchronized or matched. ACF-Net comprises Optical Flow‑Guided Motion (OFGM) to capture motion‑sensitive visual cues and suppress background noise, and Asymmetric Cross‑Modal Adaptive Fusion (ACAF) to estimate modality reliability and perform uncertainty‑aware fusion. The authors also present BirdPro, a new bird‑oriented audio‑visual benchmark with 1,919 audio recordings and 11,965 videos across 194 species, and report that ACF‑Net outperforms baselines by 2.97% in fused and 1.92% in mismatched settings.
arXiv:2607. 10299v1 Announce Type: new Abstract: Recent advances in large-scale multimodal models have drivenremarkable progress in vision-language tasks; however, comprehensiveomni-modal understanding remains under-explored, largely due to thescarcity of datasets with rich, explicitly aligned auditory cues.
arXiv:2607. 24786v1 Announce Type: cross Abstract: Weak supervision sets a practical regime for audio-visual sound source localization as dense spatial annotations are costly to obtain at scale.
The paper introduces StereoBind, a framework that links visual source motion to stereo audio generation, addressing the need for dynamic spatial correspondence in immersive applications like AR/VR and gaming. It employs three mechanisms—Visual Motion Binding, Spatial Track Encoder, and Residual Track RoPE—to coordinate visual motion with stereo sound. The authors also present StereoWorld-29K, a large-scale dataset with paired motion tracks, and StereoWorldBench for evaluating audiovisual spatial consistency, demonstrating that StereoBind improves spatial alignment while maintaining overall quality.