The paper introduces Modality Self-Play (MSP), a training framework that enables multimodal large language models to learn cross‑modal interactions without requiring paired non‑text data. MSP uses proxy‑mediated textual descriptions of unseen modalities as semantic bridges, allowing a model to generate and align these proxies with observed modalities. Experiments show that MSP achieves state‑of‑the‑art or competitive performance on several multimodal benchmarks and demonstrates effective zero‑shot modality composition through learned bridges and anchor‑supporting proxy descriptions.
By Robert Moseley, Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian, Salman Asif
arXiv:2602. 03570v2 Announce Type: replace Abstract: Audio-visual joint representation learning under Cross-Modal Generalization (CMG) aims to transfer knowledge from a labeled source modality to an unlabeled target modality through a unified discrete representation space.
By Bixing Wu, Yuhong Zhao, Zongli Ye, Jiachen Lian, Xiangyu Yue, Gopala Anumanchipalli
The paper introduces BYOAVP, a framework for Audio‑Visual Segmentation that combines adaptive contrastive learning and prototype refinement. It features a Self‑Supervised Audio Enhancement module that aligns audio with visual embeddings, and a Dynamic Prototype Constraint module that updates prototypes to improve pixel‑wise category perception and sounding‑region localization. Experiments show state‑of‑the‑art results on two benchmarks and six sub‑tasks, highlighting strong robustness and cross‑domain generalization.
By Junbo Zhang, Hang Su, Zhaofan Li, Hang Dong, Chao Sun
arXiv:2608. 16285v1 Announce Type: cross Abstract: Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues.
By Zhaojin Fu, Yuyang Hong, Qi Yang, Zili Wang, Kun Ding, Shiming Xiang, Bin Fan
arXiv:2607. 10299v1 Announce Type: new Abstract: Recent advances in large-scale multimodal models have drivenremarkable progress in vision-language tasks; however, comprehensiveomni-modal understanding remains under-explored, largely due to thescarcity of datasets with rich, explicitly aligned auditory cues.
By Kaiying Yan, Luoyi Sun, Xiao Zhou, Weidi Xie
arXiv:2606. 25225v1 Announce Type: cross Abstract: Self-supervised learning from large-scale video data has emerged as a dominant paradigm for visual representation learning.
By Revant Teotia, Adrien Bardes, Michael Rabbat, Sumit Chopra, Matthew J. Muckley, Nicolas Ballas
AVSD-Scenes is a new dataset of 12,291 audio‑visual scene descriptions for urban environments, built from the TAU Urban Audio‑Visual Scenes dataset. The descriptions are generated by first creating modality‑specific text with Qwen2‑Audio‑7B and Qwen2.5‑VL‑7B, then merging them with large language models (Qwen3‑14B, Mistral‑Small‑3.2‑24B‑Instruct‑2506, Gemma‑3‑27B‑it) to produce multimodal narratives that combine auditory and visual cues. Benchmarks show that these multimodal descriptions improve semantic alignment, cross‑modal retrieval, and scene classification accuracy (up to 95.4%) while remaining discriminative even without explicit scene labels.
By Dhanunjaya Varma Devalraju, Arshdeep Singh, Mark D. Plumbley
The paper introduces Shift-Aware Identity-guided Latent (SAIL) refinement for Referring Audio‑Visual Segmentation (Ref‑AVS), a task that requires precise segmentation of a specific object using both visual and auditory cues grounded in natural language. SAIL employs a dual‑token strategy to separate frame‑level dynamics from a global semantic anchor, a shift‑aware refinement module to correct target‑shift drift, and a decoder with continuous memory updates to preserve spatio‑temporal coherence and identity consistency. Experiments on two Ref‑AVS benchmarks show that SAIL outperforms existing AVS, Ref‑VOS, and Ref‑AVS methods.
By Kun Li, Sami Sebastian Brandt, Michael Ying Yang
The paper introduces Frequency-Aware Audio-Visual Segmentation (FAVS), a framework that treats audio-visual segmentation as a frequency-domain decomposition and recomposition task. It comprises a Frequency-Domain Enhanced Decomposer (FDED) that iteratively separates modality-specific semantics and structural features, and a Synergistic Cross-Modal Consistency (SCMC) module that uses a mixture-of-experts architecture to maintain semantic consistency and preserve modality-specific features. Experiments on three benchmark datasets show that FAVS achieves state-of-the-art performance, with qualitative visualizations supporting the effectiveness of both modules.
By Kai Peng, Yunzhe Shen, Miao Zhang, Jia Wang, Leiye Liu, Wei Ji, Jingjing Li, Yongri Piao, Huchuan Lu
The paper introduces Omni-Encoder, a single SigLIP-2 Vision Transformer that jointly processes image, video, audio, and audio‑visual data using fully shared parameters. It replaces separate modality encoders and anchor‑based alignment with a unified architecture that tokenizes audio, applies multimodal rotary position embeddings, and trains all modalities end‑to‑end through alternating‑modality batches. Omni‑Encoder achieves competitive performance on 16 benchmarks across all four modalities, outperforming individually trained counterparts, and the authors plan to release the code and models publicly.
By Xiaofei Sun, Peng Wang, Huayao Liu, Xiaoxiao Xu, Jun Song, Deqing Li, Lin Qu
arXiv:2609.21522v1 Announce Type: new
Abstract: Recent pre-trained foundation models provide rich multi-modal priors for downstream 3D vision tasks. However, the effectiveness of these representation...
By Hang Cheng, Yan Chen, Mingyu Fan, Long Zeng
arXiv:2610.14077v1 Announce Type: cross
Abstract: Embodied Audio-Visual Navigation (AVN) requires an agent to locate sound targets by jointly leveraging visual and auditory cues. While prior work has...
By Jia Li, Yinfeng Yu