← Back to all news
arXiv AI By Haoze Ma, Yidi Li, Yu Ma, Pengyuan Zhao, Yi Shi, Bin Ren, Yanwen Guo, Dimitrios Kanoulas

Which One Is Sounding? Binaural Sounding Instance Disambiguation for Audio-Visual Segmentation

Read the original on arXiv AI →

The paper introduces BiSID-AVS, a method that uses binaural audio cues to disambiguate which visual instance is actually producing sound when multiple same‑class objects are present. It grounds category‑consistent candidate regions with audio prompting, projects binaural interaural differences into a spatial response map, and refines the true sounding instance through dual‑level spatial conditioning. Experiments on binaurally extended AVS‑Synthetic and Habitat‑AVS datasets show significant gains in instance‑level mIoU and reduced false activation on non‑sounding same‑class regions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Sep 10

BinauralVAE: Spatial Audio Reconstruction For World Models

BinauralVAE is an open‑source pipeline that reconstructs spatial audio using various Variational Autoencoder architectures, including complex‑valued variants, to learn latent representations of binaural signals. The project builds on realistic acoustic data from a simulated robot navigating an environment, providing a foundation for audio‑centric world models. It aims to map the causal link between navigational actions and their acoustic outcomes, positioning sound as a complementary modality for spatial awareness.

By Luis Vitor Zerkowski, Luiz Velho
robotics
More like this →
arXiv Computer Vision
Oct 12

Shift-Aware Identity-Guided Latent Refinement for Referring Audio--Visual Segmentation

The paper introduces Shift-Aware Identity-guided Latent (SAIL) refinement for Referring Audio‑Visual Segmentation (Ref‑AVS), a task that requires precise segmentation of a specific object using both visual and auditory cues grounded in natural language. SAIL employs a dual‑token strategy to separate frame‑level dynamics from a global semantic anchor, a shift‑aware refinement module to correct target‑shift drift, and a decoder with continuous memory updates to preserve spatio‑temporal coherence and identity consistency. Experiments on two Ref‑AVS benchmarks show that SAIL outperforms existing AVS, Ref‑VOS, and Ref‑AVS methods.

By Kun Li, Sami Sebastian Brandt, Michael Ying Yang
ragcomputer-visionmultimodalbenchmarks
More like this →