arXiv Machine Learning

Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

arXiv:2607. 08839v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference.

arXiv AI

Jun 11

Information-Theoretic Decomposition for Multimodal Interaction Learning

arXiv:2606. 11614v1 Announce Type: cross Abstract: Multimodal learning hinges on capturing redundant, unique, and synergistic information across modalities, which collectively constitute multimodal interactions.

By Zequn Yang, Yake Wei, Haotian Ni, Zhihao Xu, Di Hu

fine-tuning multimodal

arXiv Machine Learning

Jun 10

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

arXiv:2606. 09853v1 Announce Type: new Abstract: A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities, and is not available from any single modality alone.

By Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen, Christos Chatzichristos, Matthew Blaschko, Maarten De Vos, Paul Pu Liang

multimodal benchmarks

Hugging Face Trending Papers

Jul 16

Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality

Cross-modal learning, i. e.

agents robotics multimodal

arXiv Machine Learning

Jun 30

Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding

arXiv:2604. 07753v2 Announce Type: replace-cross Abstract: Empowering Large Multimodal Models (LMMs) with image generation often leads to catastrophic forgetting in understanding tasks due to severe gradient conflicts.

By Xiangyue Liu, Zijian Zhang, Miles Yang, Zhao Zhong, Liefeng Bo, Ping Tan

llms diffusion multimodal benchmarks

arXiv Machine Learning

Jul 17

Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality

arXiv:2607. 14721v1 Announce Type: cross Abstract: Cross-modal learning, i.

By Kunal Pratap Singh, Ali Garjani, Rishubh Singh, Muhammad Uzair Khattak, Efe Tarhan, Jason Toskov, Andrei Atanov, O\u{g}uzhan Fatih Kar, Amir Zamir

agents robotics multimodal

arXiv Machine Learning

1d ago

UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity

arXiv:2608. 15516v1 Announce Type: new Abstract: Vision-Language Models (VLMs) have demonstrated strong performance in multimodal understanding and generation.

By Pengyu Wang, Baochen Xiong, Xiaoshan Yang, Yifan Xu, Zhang Qimeng, Haifeng Chen, Changsheng Xu

llms fine-tuning efficiency multimodal benchmarks safety

arXiv AI

Aug 11

LF${}^{2}$AR: Accounting for Layerwise Dynamics to Improve Multimodal Adaptation of Language Models

arXiv:2503. 06211v3 Announce Type: replace-cross Abstract: Text-pretrained language models (LMs) encode rich world knowledge, but adapting them to process and generate perceptual modalities such as audio and images while effectively leveraging that knowledge remains challenging.

By Santiago Cuervo, Adel Moumen, Yanis Labrak, Sameer Khurana, Antoine Laurent, Mickael Rouvier, Phil Woodland, Ricard Marxer

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 4

Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning

arXiv:2606. 04434v1 Announce Type: cross Abstract: Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for Multimodal Large Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks.

By Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh Afghah

llms efficiency multimodal benchmarks

arXiv AI

Jun 8

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

arXiv:2602. 07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions.

By Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan

llms rag fine-tuning multimodal safety

arXiv AI

Jul 14

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

arXiv:2604. 22823v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) rely on multimodal pre-training over diverse data sources, where different datasets often induce complementary cross-modal alignment capabilities.

By Zibo Shao, Baochen Xiong, Xiaoshan Yang, Yaguang Song, Qimeng Zhang, Haifeng Chen, Changsheng Xu

llms fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jul 7

LP-SFT: Local-Preserving Supervised Fine-Tuning via Multimodal Entropy Structure

arXiv:2607. 04733v1 Announce Type: cross Abstract: Supervised fine-tuning (SFT) is the standard approach for adapting pretrained language models to downstream domains, yet it often improves target-domain behavior at the cost of degrading pre-existing capabilities.

By Yueyang Wang, Baolong Bi, Shuo Lu, Jingyuan Zhang

llms fine-tuning multimodal

arXiv Machine Learning

Jul 2

Rosetta: Composable Native Multimodal Pretraining

arXiv:2607. 00293v1 Announce Type: cross Abstract: Achieving true artificial general intelligence requires foundation models capable of integrating new modalities without forgetting prior knowledge.

By Xiangyue Liu, Zijian Zhang, Miles Yang, Zhao Zhong, Liefeng Bo, Ping Tan

llms diffusion multimodal

Related stories