← Back to all news
arXiv AI By Aritra Bhowmik, Carlos Hinojosa, Fida Mohammad Thoker, Bernard Ghanem, Cees G. M. Snoek

Structured-Noise Masked Modeling for Video, Audio and Beyond

Read the original on arXiv AI →

The paper proposes a structured-noise masking technique for masked modeling in video and audio. By filtering white noise into various color noise distributions, the method creates modality‑specific masks that respect spatiotemporal and spectral characteristics without handcrafted heuristics or extra computation. Experiments demonstrate that this structured approach consistently outperforms random masking in representation learning tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 28

AudioFM: Towards General Purpose Audio Representation Learning

AudioFM is a self‑supervised framework that learns audio representations by combining recording‑level contrastive learning with patch‑wise masked latent discrimination. The model uses visible acoustic context for a student to predict masked patch representations and matches them to full‑view teacher targets from other recordings, while also incorporating weaker supervision from temporal neighbors to promote local consistency. Evaluations show that AudioFM achieves state‑of‑the‑art results across diverse audio domains in linear probing, fine‑tuning, and zero‑shot transfer, excelling in event detection, scene‑level understanding, and domain generalizability.

By Yamin Li, Ruidong Zhang, Anran Wang, Eduardo Fonseca, Dan Ellis, Zechen Li, Simon A. Lee, Tao Pang, Jake Garrison, Maxwell A. Xu, Yicheng Wang, Jitian Zhang, Weizhi Zhang, Haichao Zhang, Catie Chang, Xin Liu, Anupam Pathak, Daniel McDuff, Shwetak Patel, Jake Sunshine, Yuzhe Yang, Xiaoran Fan
fine-tuningbenchmarks
More like this →