arXiv AI

Structured-Noise Masked Modeling for Video, Audio and Beyond

The paper proposes a structured-noise masking technique for masked modeling in video and audio. By filtering white noise into various color noise distributions, the method creates modality‑specific masks that respect spatiotemporal and spectral characteristics without handcrafted heuristics or extra computation. Experiments demonstrate that this structured approach consistently outperforms random masking in representation learning tasks.

arXiv Machine Learning

Oct 28

AudioFM: Towards General Purpose Audio Representation Learning

AudioFM is a self‑supervised framework that learns audio representations by combining recording‑level contrastive learning with patch‑wise masked latent discrimination. The model uses visible acoustic context for a student to predict masked patch representations and matches them to full‑view teacher targets from other recordings, while also incorporating weaker supervision from temporal neighbors to promote local consistency. Evaluations show that AudioFM achieves state‑of‑the‑art results across diverse audio domains in linear probing, fine‑tuning, and zero‑shot transfer, excelling in event detection, scene‑level understanding, and domain generalizability.

By Yamin Li, Ruidong Zhang, Anran Wang, Eduardo Fonseca, Dan Ellis, Zechen Li, Simon A. Lee, Tao Pang, Jake Garrison, Maxwell A. Xu, Yicheng Wang, Jitian Zhang, Weizhi Zhang, Haichao Zhang, Catie Chang, Xin Liu, Anupam Pathak, Daniel McDuff, Shwetak Patel, Jake Sunshine, Yuzhe Yang, Xiaoran Fan

fine-tuning benchmarks

Hugging Face Trending Papers

Oct 25

AudioFM: Towards General Purpose Audio Representation Learning

Learning audio representations that effectively bridge holistic global soundscapes with fine-grained acoustic moments is a long-standing goal in self-supervised learning. Recording-level contrastive o...

fine-tuning benchmarks

arXiv Machine Learning

Sep 10

A Theoretical Framework for Masked Pretraining (MPT)

arXiv:2609.06460v1 Announce Type: new Abstract: Recently, Masked Pretraining (MPT) based on reconstruction pretraining tasks has risen to a promising self-supervised learning paradigm across various...

By Qi Zhang, Runyu Zhou, Yifei Wang, Yisen Wang

fine-tuning safety

arXiv Computer Vision

Aug 25

MLLM-Assisted Audio VOS: A 3rd Place Report for the MeViS-Audio Track, 8th LSVOS Challenge

arXiv:2608.23234v1 Announce Type: new Abstract: In this technical report, we present a training-free framework for audio-guided video object segmentation, which integrates Multimodal Large Language M...

By Liangtao Shi, Jinxia Xie, Xiantao Hu, Ting Liu

llms computer-vision fine-tuning multimodal

arXiv Computer Vision

Oct 1

Image Classifiers are Efficient Self-Supervised Video Representation Learners

arXiv:2609.40347v1 Announce Type: new Abstract: We introduce VideoMSN, a Masked Siamese Network framework for efficient self-supervised spatio-temporal representation learning in videos. Instead of r...

By Owais Iqbal, Sudipta Sarkar, Shyam Marjit, Omprakash Chakraborty, Anirban Chakraborty, Abir Das

llms rag computer-vision benchmarks

arXiv AI

Sep 30

AVIO: Learning to Add and Remove Sounding Objects in Audiovisual Scenes

arXiv:2609.36503v1 Announce Type: new Abstract: Adding or removing a sounding object requires coordinated changes to visual content and sound while preserving the surrounding scene. Yet paired superv...

By Weihan Xu, Kan Jen Cheng, Koichi Saito, Jingyu Shi, Tingle Li, Yisi Liu, Liming Wang, Masato Ishii, Takashi Shibuya, Gopala Anumanchipalli, Paul Pu Liang

multimodal

arXiv AI

Jul 20

AV-JEPA: Extending LeJEPA to Audio-Visual Self-Supervised Learning

arXiv:2607. 15295v1 Announce Type: cross Abstract: We present AV-JEPA, an elegant multimodal extension of LeJEPA to audio-visual self-supervised learning.

By Benjamin Robson, Santeri Mentu, Wenshuai Zhao, Arno Solin

llms rag diffusion computer-vision multimodal safety

arXiv Machine Learning

Jun 16

From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation

arXiv:2606. 14791v1 Announce Type: cross Abstract: Self-supervised learning advances audio representation for multimedia analysis.

By Fengrui Liu, Ruiyang Huang, Qijian Zheng, Yuanfang Wang, Feng Liu

llms diffusion benchmarks safety

arXiv AI

Jun 11

Feature-Aligned Speech Watermarking for Robustness to Reconstruction Distortions

arXiv:2606. 11828v1 Announce Type: cross Abstract: Audio watermarking aims to embed identifiable information into audio while remaining imperceptible.

By Haiyun Li, Shuhai Peng, Zhisheng Zhang, Jingran Xie, Xiaofeng Xie, Hanyang Peng, Zhiyong Wu

rag

arXiv Machine Learning

Aug 18

Prototype-Rectified Iterative Self-supervised Manifold Denoising under Severe Acoustic Shift

arXiv:2608. 15037v1 Announce Type: cross Abstract: Audio-Text Foundation Models (ATMs) fail catastrophically under severe acoustic noise, yet existing adaptation strategies either rely on gradient-based Test-Time Adaptation (TTA), which reinforces noise rather than signal, or on prompt tuning that requires privileged noise annotations unavailable at inference.

By Ashish Anand Shukla, Rini Smita Thakur, Aryan Das, Vinod K. Kurmi

diffusion multimodal safety

arXiv AI

Jul 1

BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations

arXiv:2606. 30700v1 Announce Type: cross Abstract: Self-supervised learning enables audio representations that transfer across domains and tasks.

By Ludovic K. Tuncay (IRIT-SAMoVA), Etienne Labb\'e (IRIT-SAMoVA), Thomas Pellegrini (IRIT-SAMoVA)

llms benchmarks

arXiv Machine Learning

Aug 17

SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from Videos

arXiv:2510. 02916v2 Announce Type: replace-cross Abstract: We propose SALSA-V, a multimodal video-to-audio generation model capable of synthesizing highly synchronized, high-fidelity long-form audio from silent video content.

By Amir Dellali, Luca A. Lanzend\"orfer, Florian Gr\"otschla, Roger Wattenhofer

diffusion fine-tuning multimodal benchmarks safety

Related stories