arXiv Machine Learning
Multimodal Information Interactions Bottleneck for Multimodal Self-Supervised Learning
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jun 11
Information-Theoretic Decomposition for Multimodal Interaction Learning
arXiv:2606. 11614v1 Announce Type: cross Abstract: Multimodal learning hinges on capturing redundant, unique, and synergistic information across modalities, which collectively constitute multimodal interactions.
arXiv AI
Oct 9
HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling
The paper introduces HRIL, a method for learning multimodal representations that explicitly models higher‑order interactions among modalities using a cross‑moment tensor and Tucker decomposition. It adds a synergy‑aware regularizer to preserve higher‑order coupling capacity, aiming to capture synergistic signals that only emerge from joint configurations of multiple modalities. Experiments on controlled synergy tasks and real‑world benchmarks show consistent improvements over existing multimodal contrastive methods, especially on tasks dominated by synergistic interactions.
arXiv Machine Learning
Sep 22
Generalized Multimodal Foundation Model
The paper introduces a generalized multimodal foundation model that can handle arbitrary combinations of modalities and prediction tasks. It trains on large-scale synthetic multimodal datasets with diverse causal structures to learn transferable multimodal correlations. Experiments on 18 real-world datasets across 12 modalities and 11 tasks show competitive performance compared to specialized models without task-specific adaptation.
arXiv AI
Oct 16
MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data
The paper introduces Modality Self-Play (MSP), a training framework that enables multimodal large language models to learn cross‑modal interactions without requiring paired non‑text data. MSP uses proxy‑mediated textual descriptions of unseen modalities as semantic bridges, allowing a model to generate and align these proxies with observed modalities. Experiments show that MSP achieves state‑of‑the‑art or competitive performance on several multimodal benchmarks and demonstrates effective zero‑shot modality composition through learned bridges and anchor‑supporting proxy descriptions.
arXiv Machine Learning
Jul 2
Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning
arXiv:2505. 19614v2 Announce Type: replace Abstract: Multimodal learning has seen remarkable progress, particularly with large-scale pre-training across various modalities.
arXiv AI
Oct 15
Dependency, Compression, and Synergy: A Unified Information-Theoretic View of Multimodal Learning
arXiv:2609.14421v1 Announce Type: cross Abstract: Recent advances in multimodal foundation models have intensified the need to understand how different modalities share, preserve, and complement info...