arXiv Machine Learning
Jul 17
Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality
arXiv:2607. 14721v1 Announce Type: cross Abstract: Cross-modal learning, i.
Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality
Cross-modal learning, i. e.
Related stories
arXiv Machine Learning
Jul 13
Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing
arXiv:2607. 08839v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference.
arXiv Machine Learning
Sep 22
Generalized Multimodal Foundation Model
The paper introduces a generalized multimodal foundation model that can handle arbitrary combinations of modalities and prediction tasks. It trains on large-scale synthetic multimodal datasets with diverse causal structures to learn transferable multimodal correlations. Experiments on 18 real-world datasets across 12 modalities and 11 tasks show competitive performance compared to specialized models without task-specific adaptation.
arXiv Computer Vision
Sep 22
Cognitive Action Reasoning for Proactive Robots from Human-Centered Multimodal Observations
The paper introduces ProAction, a multimodal dataset of 10,000 samples comprising visual, audio, and text inputs across 12 daily-life scenarios, designed to support the Proactive Robot Action Reasoning (ProRobo) problem. It presents a two-stage human-in-the-loop annotation pipeline that incorporates appraisal and Theory-of-Mind considerations to generate cognitively grounded high-level action labels. The authors benchmark multimodal large language models and propose MMC2Act, showing that training on ProAction significantly improves proactive action reasoning compared to general-purpose models.
arXiv Computation and Language
3d ago
Fusion Anything: A Generalized Multimodal Foundation Model
The paper introduces Fusion Anything Model (FAM), a foundation model designed for generalized multimodal data fusion that can handle arbitrary modality combinations and prediction tasks. FAM is trained on large-scale synthetic multimodal datasets generated via Structural Multimodal Causal Models (SMCMs), enabling it to encode transferable multimodal correlations. Experiments on 18 real-world datasets across 12 modalities and 11 tasks show that FAM performs competitively with specialized models without requiring task-specific adaptation.
arXiv Machine Learning
Jul 2
Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning
arXiv:2505. 19614v2 Announce Type: replace Abstract: Multimodal learning has seen remarkable progress, particularly with large-scale pre-training across various modalities.
arXiv Machine Learning
Sep 15
TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals
TwinICL is a procedurally generated benchmark that pairs matched text and image versions of tasks to enable controlled comparison of in‑context learning (ICL) across modalities. Experiments on six open‑weight models and 38 tasks show that multimodal ICL consistently underperforms text‑only ICL, with varying gaps by task family. Interventions targeting visual access, task framing, and reasoning can recover strong multimodal performance on a diagnostic subset, yet a modality gap remains even when explicit task instructions are provided, highlighting the dual role of demonstrations as context and evidence.
Hugging Face Trending Papers
4d ago
UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement
Modern multimodal models bring generation and understanding into a single unified system, which enables them to provide and learn from their own feedback. Motivated by this unified capacity, we introd...
arXiv AI
2d ago
FairSSL: Fair Multimodal Self-Supervised Learning
arXiv:2508.16748v2 Announce Type: replace-cross Abstract: Prevalent multimodal self-supervised learning (SSL) methods rely on the redundancy assumption: that different views share substantial task-re...
arXiv AI
3d ago
UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement
UniEvo‑VL is a self‑evolving framework that lets multimodal models improve themselves by using their own critiques as privileged information. The method trains a single model to act as both teacher and student, minimizing divergence between their diffusion distributions over sampling trajectories. Experiments on Qwen‑image‑2512 show significant gains in image generation metrics, and stronger external critics further raise the improvement ceiling, though results vary across tasks.
arXiv Machine Learning
Sep 24
Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classification
The paper identifies that in multimodal learning, optimization often produces asymmetric certainty gains, with the stronger modality becoming more confident than the weaker one, which leads to imbalanced contributions and suboptimal performance. The authors attribute this issue to unimodal characteristics and propose a Max Confidence Regularization (MaxCR) method that tracks each modality’s semantic confidence via a nonlinear sparsity measure and applies max suppression and excitation to balance confidence levels. Experiments on standard datasets demonstrate that MaxCR improves overall performance compared to state‑of‑the‑art multimodal baselines.
arXiv AI
Jul 7
K9-Bench: Evaluating Multimodal LLMs on Canine-Centric Videos
arXiv:2607. 02680v1 Announce Type: cross Abstract: MLLMs have shown strong zero-shot capabilities across diverse inputs such as across images, video, audio, and text.