arXiv Machine Learning
Jun 16
SPICE: Synergy and Partial Information Based Curriculum Evolution
arXiv:2606. 16639v1 Announce Type: new Abstract: Multimodal learning exploits complementary information across heterogeneous modalities.
Information-Theoretic Decomposition for Multimodal Interaction Learning
arXiv:2606. 11614v1 Announce Type: cross Abstract: Multimodal learning hinges on capturing redundant, unique, and synergistic information across modalities, which collectively constitute multimodal interactions.
Related stories
arXiv Machine Learning
Jul 2
Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning
arXiv:2505. 19614v2 Announce Type: replace Abstract: Multimodal learning has seen remarkable progress, particularly with large-scale pre-training across various modalities.
arXiv Machine Learning
Jul 13
Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing
arXiv:2607. 08839v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference.
arXiv AI
Jun 2
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
arXiv:2606. 00959v1 Announce Type: new Abstract: Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment.
arXiv Machine Learning
Jun 10
SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning
arXiv:2606. 09853v1 Announce Type: new Abstract: A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities, and is not available from any single modality alone.
arXiv Machine Learning
Jul 2
Rosetta: Composable Native Multimodal Pretraining
arXiv:2607. 00293v1 Announce Type: cross Abstract: Achieving true artificial general intelligence requires foundation models capable of integrating new modalities without forgetting prior knowledge.
arXiv Machine Learning
Jun 30
Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding
arXiv:2604. 07753v2 Announce Type: replace-cross Abstract: Empowering Large Multimodal Models (LMMs) with image generation often leads to catastrophic forgetting in understanding tasks due to severe gradient conflicts.
arXiv Machine Learning
Aug 5
DAIF: A Data-Driven Intermediate Fusion Framework for Multimodal Supervised Learning via Approximate Message Passing
arXiv:2608. 02769v1 Announce Type: cross Abstract: Multimodal supervised learning seeks to leverage multiple heterogeneous data sources to improve predictive performance.
arXiv AI
Aug 6
C$^2$MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learning
arXiv:2608. 04013v1 Announce Type: cross Abstract: Recent advances in Multimodal Emotion Recognition in Conversations (MERC) highlight its reliance on complete multimodal inputs.
arXiv Machine Learning
Jul 7
Beyond Modality Fusion: Deep Ensembles for Multimodal Classification
arXiv:2607. 05019v1 Announce Type: new Abstract: In multimodal classification, late-fusion approaches classify concatenated modality-specific features extracted by unimodal neural networks.
arXiv Machine Learning
Jul 21
MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning
arXiv:2607. 16789v1 Announce Type: new Abstract: Real-world perception and decision making are inherently multimodal, integrating complementary signals across modalities.
arXiv Machine Learning
Jun 4
RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities
arXiv:2606. 05109v1 Announce Type: new Abstract: To leverage the full potential of multimodal data, we need representations that go beyond the state-of-the-art alignment and fusion approaches and exploit all cross-modal interactions without sacrificing modality-specific information.