AudioFM is a self‑supervised framework that learns audio representations by combining recording‑level contrastive learning with patch‑wise masked latent discrimination. The model uses visible acoustic context for a student to predict masked patch representations and matches them to full‑view teacher targets from other recordings, while also incorporating weaker supervision from temporal neighbors to promote local consistency. Evaluations show that AudioFM achieves state‑of‑the‑art results across diverse audio domains in linear probing, fine‑tuning, and zero‑shot transfer, excelling in event detection, scene‑level understanding, and domain generalizability.
By Yamin Li, Ruidong Zhang, Anran Wang, Eduardo Fonseca, Dan Ellis, Zechen Li, Simon A. Lee, Tao Pang, Jake Garrison, Maxwell A. Xu, Yicheng Wang, Jitian Zhang, Weizhi Zhang, Haichao Zhang, Catie Chang, Xin Liu, Anupam Pathak, Daniel McDuff, Shwetak Patel, Jake Sunshine, Yuzhe Yang, Xiaoran Fan
arXiv:2608. 19863v1 Announce Type: cross Abstract: Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance.
By Umberto Cappellazzo, Xubo Liu, Stavros Petridis, Maja Pantic
arXiv:2609.23152v1 Announce Type: cross
Abstract: Recently proposed self-supervised audio encoders learn powerful general-purpose representations of sound scenes, yet they are spatially blind. To sup...
By Goksenin Yuksel, Marcel van Gerven, Kiki van der Heijden
arXiv:2606. 29901v1 Announce Type: cross Abstract: Sound event detection (SED) is a core module for acoustic environmental analysis, yet its performance is often limited by scarce labeled data.
By Nian Shao, Xian Li, Xiaofei Li
Self-supervised learning from large-scale video data has emerged as a dominant paradigm for visual representation learning. Since audio and visual streams naturally co-occur in video data, extending this success to jointly learn from both modalities is a natural next step, yet it remains challenging.
arXiv:2606. 30700v1 Announce Type: cross Abstract: Self-supervised learning enables audio representations that transfer across domains and tasks.
By Ludovic K. Tuncay (IRIT-SAMoVA), Etienne Labb\'e (IRIT-SAMoVA), Thomas Pellegrini (IRIT-SAMoVA)
The paper proposes a structured-noise masking technique for masked modeling in video and audio. By filtering white noise into various color noise distributions, the method creates modality‑specific masks that respect spatiotemporal and spectral characteristics without handcrafted heuristics or extra computation. Experiments demonstrate that this structured approach consistently outperforms random masking in representation learning tasks.
By Aritra Bhowmik, Carlos Hinojosa, Fida Mohammad Thoker, Bernard Ghanem, Cees G. M. Snoek
The rapid advancement of generative AI models is leading to more realistic deepfake media, encompassing the manipulation of audio, video, or both. This raises severe privacy and societal concerns.
arXiv:2606. 15117v1 Announce Type: cross Abstract: The rapid advancement of generative AI models is leading to more realistic deepfake media, encompassing the manipulation of audio, video, or both.
By Elham Abolhasani, Maryam Ramezani, Hamid R. Rabiee
arXiv:2607. 15295v1 Announce Type: cross Abstract: We present AV-JEPA, an elegant multimodal extension of LeJEPA to audio-visual self-supervised learning.
By Benjamin Robson, Santeri Mentu, Wenshuai Zhao, Arno Solin
arXiv:2606. 14791v1 Announce Type: cross Abstract: Self-supervised learning advances audio representation for multimedia analysis.
By Fengrui Liu, Ruiyang Huang, Qijian Zheng, Yuanfang Wang, Feng Liu
arXiv:2610.27491v1 Announce Type: cross
Abstract: Real-world audio typically contains concurrent sound events, yet standard self-supervised methods treat each clip as an isolated training unit withou...
By Kohei Yamamoto, Kosuke Okusa