The paper introduces a framework to tackle modality imbalance in multimodal learning by focusing on sample-level variations. It defines a Modality Gap metric to measure prediction discrepancies, models the resulting bimodal distribution with a Gaussian Mixture Model, and uses Bayesian probabilities for soft separation of balanced and imbalanced samples. A two‑stage training process—Warm‑up and Adaptive Training—reallocates loss weights based on the GMM, strengthening alignment for imbalanced samples while favoring fusion for balanced ones, and shows superior performance over existing baselines.
arXiv:2607. 05019v1 Announce Type: new Abstract: In multimodal classification, late-fusion approaches classify concatenated modality-specific features extracted by unimodal neural networks.
arXiv:2608. 02769v1 Announce Type: cross Abstract: Multimodal supervised learning seeks to leverage multiple heterogeneous data sources to improve predictive performance.
arXiv:2607. 27289v1 Announce Type: new Abstract: The promise of multimodal fusion lies in combining complementary sources of evidence, yet more evidence does not always yield a better prediction.
arXiv:2606. 11190v1 Announce Type: new Abstract: Cross-modal alignment (CA) and cross-modal prediction (CP) are the dominant paradigms for multimodal representation learning, yet there is no systematic understanding of when each succeeds, when each fails, and when cross-modal training helps at all -- a gap that leaves practitioners, especially in scientific domains like biomedicine or astrophysics, with heterogeneous instruments and multiple levels of organization and measurement, unable to diagnose why standard methods underperform the best single modality.
Cross-modal alignment (CA) and cross-modal prediction (CP) are the dominant paradigms for multimodal representation learning, yet there is no systematic understanding of when each succeeds, when each fails, and when cross-modal training helps at all -- a gap that leaves practitioners, especially in scientific domains like biomedicine or astrophysics, with heterogeneous instruments and multiple levels of organization and measurement, unable to diagnose why standard methods underperform the best single modality. We develop a unified linear framework that addresses both questions.
arXiv:2606. 15743v1 Announce Type: new Abstract: This paper addresses the missing-modality challenge in multi-modal learning by introducing Unsupervised Learning for Missing Modalities in Multi-Modal Learning (UL4M4), a flexible framework that imputes missing feature embeddings in a task-independent manner before supervised prediction.
arXiv:2606. 02679v1 Announce Type: new Abstract: Multimodal systems often benefit from combining information across language, sound, and visual streams, but this benefit is not guaranteed.
arXiv:2602. 07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions.
arXiv:2601.21670v4 Announce Type: replace-cross Abstract: Multimodal fusion is often treated as an optimization-balancing problem, where training signals are adjusted to prevent one modality from dom...
arXiv:2608. 07183v1 Announce Type: new Abstract: Multimodal fusion architectures typically assume all modalities are available at inference, yet sensor failures, acquisition variability, and cost constraints routinely produce incomplete observations.
The paper introduces CAT‑GS, a training controller that stabilizes multimodal neural networks by addressing three failure modes: modality imbalance, unstable gating, and fusion interference. CAT‑GS calibrates teacher-derived reliability, applies a margin‑thresholded gating policy, caps gradient budgets, and uses fusion‑only PCGrad, all without altering model architectures or losses. Experiments on audio‑visual, tri‑modal, synthetic, and cross‑domain benchmarks show that CAT‑GS matches or surpasses strong imbalance‑aware baselines while producing smoother gating and fewer conflicting fusion gradients.
arXiv:2510. 07328v2 Announce Type: replace-cross Abstract: Medical decision systems increasingly rely on data from multiple sources to ensure reliable and unbiased diagnosis.