arXiv Machine Learning

MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning

arXiv:2606. 25225v1 Announce Type: cross Abstract: Self-supervised learning from large-scale video data has emerged as a dominant paradigm for visual representation learning.

Hugging Face Trending Papers

Jun 23

MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning

Self-supervised learning from large-scale video data has emerged as a dominant paradigm for visual representation learning. Since audio and visual streams naturally co-occur in video data, extending this success to jointly learn from both modalities is a natural next step, yet it remains challenging.

rag multimodal benchmarks

arXiv AI

Jul 20

AV-JEPA: Extending LeJEPA to Audio-Visual Self-Supervised Learning

arXiv:2607. 15295v1 Announce Type: cross Abstract: We present AV-JEPA, an elegant multimodal extension of LeJEPA to audio-visual self-supervised learning.

By Benjamin Robson, Santeri Mentu, Wenshuai Zhao, Arno Solin

llms rag diffusion computer-vision multimodal safety

arXiv AI

Jun 6

Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning

arXiv:2606. 05173v1 Announce Type: cross Abstract: Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT, yet it encourages representations that are strongly anchored to surface-form token identity rather than deeper semantic structure.

By Aimen Boukhari

rag diffusion benchmarks

arXiv Machine Learning

Jun 16

MVEB: Massive Video Embedding Benchmark

arXiv:2606. 14958v1 Announce Type: cross Abstract: We introduce the Massive Video Embedding Benchmark (MVEB), a 23-task benchmark for video embeddings spanning classification, zero-shot classification, clustering, pair classification, retrieval, and video-centric question answering.

By Adnan El Assadi, Roman Solomatin, Isaac Chung, Chenghao Xiao, Deep Shah, Manan Dey, Shriya Sudhakar, Zacharie Bugaud, Wissam Siblini, Ayush Sunil Munot, Yashwanth Devavarapu, Rakshitha Ireddi, Michelle Yang, M\'arton Kardos, Niklas Muennighoff, Kenneth Enevoldsen

rag nlp multimodal benchmarks

arXiv Machine Learning

Jul 14

Empowering Long-form Omni-modal Understanding with Robust Audio Perception

arXiv:2607. 10299v1 Announce Type: new Abstract: Recent advances in large-scale multimodal models have drivenremarkable progress in vision-language tasks; however, comprehensiveomni-modal understanding remains under-explored, largely due to thescarcity of datasets with rich, explicitly aligned auditory cues.

By Kaiying Yan, Luoyi Sun, Xiao Zhou, Weidi Xie

fine-tuning multimodal benchmarks

arXiv Machine Learning

Aug 4

Asleep at the Wheel: JEPA's Limitations in Evaluating Novel Driving Data

arXiv:2608. 01336v1 Announce Type: cross Abstract: Modern autonomous-driving fleets record far more video than human reviewers can inspect.

By Advait Pavuluri, Shamik Karkhanis, Uzma Mushtaque

rag agents fine-tuning benchmarks

arXiv Machine Learning

Jul 28

Asymmetric Hierarchical Anchoring for Robust Audio-Visual Cross-Modal Generalization

arXiv:2602. 03570v2 Announce Type: replace Abstract: Audio-visual joint representation learning under Cross-Modal Generalization (CMG) aims to transfer knowledge from a labeled source modality to an unlabeled target modality through a unified discrete representation space.

By Bixing Wu, Yuhong Zhao, Zongli Ye, Jiachen Lian, Xiangyu Yue, Gopala Anumanchipalli

efficiency multimodal benchmarks safety

arXiv AI

Jul 10

Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks

arXiv:2607. 07907v1 Announce Type: cross Abstract: With the growing adoption of VLMs, DMs, LLMs, and AFMs, these multimodal foundation models can inadvertently encode sensitive, copyrighted, biased, or unsafe cross-modal associations that originate from their training data.

By Nobin Sarwar, Shubhashis Roy Dipta, Zheyuan Liu, Vaidehi Patil

llms multimodal benchmarks

arXiv Machine Learning

Aug 6

Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation

arXiv:2608. 04902v1 Announce Type: cross Abstract: Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis.

By Zehua Chen, Junyou Wang, Yuxuan Jiang, Zhenying Fang, Yusheng Dai, Jianfei Chen, Ziwei Liu, Jun Zhu

diffusion multimodal benchmarks

arXiv AI

Aug 11

Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models

arXiv:2608. 09227v1 Announce Type: new Abstract: Omnimodal language models (OLMs) enable unified audio-visual understanding, but processing long joint token sequences makes inference computationally prohibitive.

By Puneet Mathur, Manan Suri, Dinesh Manocha

llms nlp fine-tuning efficiency multimodal benchmarks

arXiv Machine Learning

Jun 4

RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities

arXiv:2606. 05109v1 Announce Type: new Abstract: To leverage the full potential of multimodal data, we need representations that go beyond the state-of-the-art alignment and fusion approaches and exploit all cross-modal interactions without sacrificing modality-specific information.

By Vasiliki Rizou, Pascal Frossard, Dorina Thanou

rag multimodal benchmarks safety

arXiv Machine Learning

Jul 21

MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning

arXiv:2607. 16789v1 Announce Type: new Abstract: Real-world perception and decision making are inherently multimodal, integrating complementary signals across modalities.

By Sana Tonekaboni, Viktoria Schuster, Caroline Uhler

fine-tuning multimodal benchmarks safety

Related stories