AudioFM is a self‑supervised framework that learns audio representations by combining recording‑level contrastive learning with patch‑wise masked latent discrimination. The model uses visible acoustic context for a student to predict masked patch representations and matches them to full‑view teacher targets from other recordings, while also incorporating weaker supervision from temporal neighbors to promote local consistency. Evaluations show that AudioFM achieves state‑of‑the‑art results across diverse audio domains in linear probing, fine‑tuning, and zero‑shot transfer, excelling in event detection, scene‑level understanding, and domain generalizability.
Learning audio representations that effectively bridge holistic global soundscapes with fine-grained acoustic moments is a long-standing goal in self-supervised learning. Recording-level contrastive o...
arXiv:2608. 19863v1 Announce Type: cross Abstract: Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance.
arXiv:2606. 30700v1 Announce Type: cross Abstract: Self-supervised learning enables audio representations that transfer across domains and tasks.
arXiv:2609.23152v1 Announce Type: cross Abstract: Recently proposed self-supervised audio encoders learn powerful general-purpose representations of sound scenes, yet they are spatially blind. To sup...
arXiv:2606. 29901v1 Announce Type: cross Abstract: Sound event detection (SED) is a core module for acoustic environmental analysis, yet its performance is often limited by scarce labeled data.
arXiv:2607. 15295v1 Announce Type: cross Abstract: We present AV-JEPA, an elegant multimodal extension of LeJEPA to audio-visual self-supervised learning.
Self-supervised learning from large-scale video data has emerged as a dominant paradigm for visual representation learning. Since audio and visual streams naturally co-occur in video data, extending this success to jointly learn from both modalities is a natural next step, yet it remains challenging.
arXiv:2606. 15117v1 Announce Type: cross Abstract: The rapid advancement of generative AI models is leading to more realistic deepfake media, encompassing the manipulation of audio, video, or both.
The paper proposes a structured-noise masking technique for masked modeling in video and audio. By filtering white noise into various color noise distributions, the method creates modality‑specific masks that respect spatiotemporal and spectral characteristics without handcrafted heuristics or extra computation. Experiments demonstrate that this structured approach consistently outperforms random masking in representation learning tasks.
arXiv:2610.27491v1 Announce Type: cross Abstract: Real-world audio typically contains concurrent sound events, yet standard self-supervised methods treat each clip as an isolated training unit withou...
arXiv:2606. 14791v1 Announce Type: cross Abstract: Self-supervised learning advances audio representation for multimedia analysis.
The rapid advancement of generative AI models is leading to more realistic deepfake media, encompassing the manipulation of audio, video, or both. This raises severe privacy and societal concerns.