← Back to all news
arXiv Computer Vision By Junbo Zhang, Hang Su, Zhaofan Li, Hang Dong, Chao Sun

Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation

Read the original on arXiv Computer Vision →

The paper introduces BYOAVP, a framework for Audio‑Visual Segmentation that combines adaptive contrastive learning and prototype refinement. It features a Self‑Supervised Audio Enhancement module that aligns audio with visual embeddings, and a Dynamic Prototype Constraint module that updates prototypes to improve pixel‑wise category perception and sounding‑region localization. Experiments show state‑of‑the‑art results on two benchmarks and six sub‑tasks, highlighting strong robustness and cross‑domain generalization.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computer Vision
Oct 21

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

The paper introduces Frequency-Aware Audio-Visual Segmentation (FAVS), a framework that treats audio-visual segmentation as a frequency-domain decomposition and recomposition task. It comprises a Frequency-Domain Enhanced Decomposer (FDED) that iteratively separates modality-specific semantics and structural features, and a Synergistic Cross-Modal Consistency (SCMC) module that uses a mixture-of-experts architecture to maintain semantic consistency and preserve modality-specific features. Experiments on three benchmark datasets show that FAVS achieves state-of-the-art performance, with qualitative visualizations supporting the effectiveness of both modules.

By Kai Peng, Yunzhe Shen, Miao Zhang, Jia Wang, Leiye Liu, Wei Ji, Jingjing Li, Yongri Piao, Huchuan Lu
computer-visionmultimodalbenchmarks
More like this →
arXiv Computer Vision
Oct 12

Shift-Aware Identity-Guided Latent Refinement for Referring Audio--Visual Segmentation

The paper introduces Shift-Aware Identity-guided Latent (SAIL) refinement for Referring Audio‑Visual Segmentation (Ref‑AVS), a task that requires precise segmentation of a specific object using both visual and auditory cues grounded in natural language. SAIL employs a dual‑token strategy to separate frame‑level dynamics from a global semantic anchor, a shift‑aware refinement module to correct target‑shift drift, and a decoder with continuous memory updates to preserve spatio‑temporal coherence and identity consistency. Experiments on two Ref‑AVS benchmarks show that SAIL outperforms existing AVS, Ref‑VOS, and Ref‑AVS methods.

By Kun Li, Sami Sebastian Brandt, Michael Ying Yang
ragcomputer-visionmultimodalbenchmarks
More like this →
arXiv Computer Vision
Aug 27

Asymmetric Cross-Modal Fine-Grained Visual Categorization: ACF-Net and the BirdPro Benchmark

The paper introduces ACF-Net, an optical flow‑guided framework for asymmetric audio‑visual fine‑grained visual categorization (FGVC), addressing challenges where video and audio are not strictly synchronized or matched. ACF-Net comprises Optical Flow‑Guided Motion (OFGM) to capture motion‑sensitive visual cues and suppress background noise, and Asymmetric Cross‑Modal Adaptive Fusion (ACAF) to estimate modality reliability and perform uncertainty‑aware fusion. The authors also present BirdPro, a new bird‑oriented audio‑visual benchmark with 1,919 audio recordings and 11,965 videos across 194 species, and report that ACF‑Net outperforms baselines by 2.97% in fused and 1.92% in mismatched settings.

By Bohan Deng, Shuo Ye, Zitong Yu
computer-visionmultimodalbenchmarkssafety
More like this →