arXiv AI

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

arXiv:2604. 22823v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) rely on multimodal pre-training over diverse data sources, where different datasets often induce complementary cross-modal alignment capabilities.

arXiv AI

Jun 8

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

arXiv:2602. 07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions.

By Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan

llms rag fine-tuning multimodal safety

arXiv AI

Jul 3

SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment

arXiv:2511. 01390v2 Announce Type: replace-cross Abstract: Fine-grained cross-modal alignment aims to establish precise local correspondences between vision and language, forming a cornerstone for visual question answering and related multimodal applications.

By Xinyu Mao, Junsi Li, Haoji Zhang, Yu Liang, Ming Sun

llms diffusion nlp multimodal safety

arXiv AI

Jun 17

Plug-and-Adapt: Multimodal Coreference Resolution at First Sight with a Pretrained Alignment Model

arXiv:2606. 17950v1 Announce Type: cross Abstract: Visual information helps resolve ambiguity in coreference resolution, leading to notable performance gains.

By Jinghan Wu, Jing Li, Ivor W. Tsang, Xuetao Zhang

multimodal benchmarks safety

arXiv AI

Jul 16

Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping

arXiv:2508. 12466v2 Announce Type: replace-cross Abstract: Traditional multimodal learning approaches rely on alignment pre-training to bridge vision and language modalities, typically by projecting visual features into discrete text token spaces using large-scale image--text data.

By Xuhui Zhan, Tyler Derr

llms rag multimodal benchmarks safety

arXiv Machine Learning

Jul 31

MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understanding

arXiv:2511. 12449v3 Announce Type: replace-cross Abstract: Recent Multimodal Large Language Models (MLLMs) have significantly advanced e-commerce product understanding.

By Zhanheng Nie, Chenghan Fu, Daoze Zhang, Junxian Wu, Wanxian Guan, Pengjie Wang, Jian Xu, Bo Zheng

llms multimodal benchmarks safety data-engineering

arXiv AI

Jun 30

Multimodal Representation Alignment for Cross-modal Information Retrieval

arXiv:2506. 08774v2 Announce Type: replace-cross Abstract: Different machine learning models can represent the same underlying concept in different ways.

By Fan Xu, Luis A. Leiva

llms rag multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning

arXiv:2606. 01207v1 Announce Type: cross Abstract: The choice between cross-attention and concatenation for multimodal fusion remains governed by practitioner intuition rather than principled understanding.

By Zhiqiang Zhou, Xuezhen Xie

llms multimodal safety

arXiv AI

Aug 11

MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks

arXiv:2507. 19634v4 Announce Type: replace-cross Abstract: Recent advances in large language models have laid the foundation for multimodal LLMs (MLLMs), which unify text, speech, and vision within a single framework.

By Sara Papi, Maike Z\"ufle, Marco Gaido, Beatrice Savoldi, Danni Liu, Ioannis Douros, Luisa Bentivogli, Jan Niehues

llms nlp multimodal benchmarks

arXiv AI

Jul 31

Progressive Multimodal Alignment for Continual Instruction Tuning

arXiv:2607. 26947v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) rely on a projector to align visual representations with the language embedding space, making it central to cross-modal understanding.

By Duzhen Zhang, Yahan Yu, Qiaoyi Su, Jiahua Dong, Tielin Zhang

llms rag fine-tuning multimodal benchmarks safety

arXiv AI

Aug 11

LF${}^{2}$AR: Accounting for Layerwise Dynamics to Improve Multimodal Adaptation of Language Models

arXiv:2503. 06211v3 Announce Type: replace-cross Abstract: Text-pretrained language models (LMs) encode rich world knowledge, but adapting them to process and generate perceptual modalities such as audio and images while effectively leveraging that knowledge remains challenging.

By Santiago Cuervo, Adel Moumen, Yanis Labrak, Sameer Khurana, Antoine Laurent, Mickael Rouvier, Phil Woodland, Ricard Marxer

llms multimodal benchmarks safety

arXiv Machine Learning

Jul 2

Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning

arXiv:2505. 19614v2 Announce Type: replace Abstract: Multimodal learning has seen remarkable progress, particularly with large-scale pre-training across various modalities.

By Sanghyuk Chun, Olga Russakovsky

multimodal benchmarks safety

arXiv Machine Learning

Jun 15

Context-aware Modality-Topology Co-Alignment for Multimodal Attributed Graphs

arXiv:2606. 14172v1 Announce Type: new Abstract: Multimodal Attributed Graphs (MAGs) model real-world entities by coupling graph topology with heterogeneous attributes such as text and images.

By Sirui Zhang, Xu Wang, Zhengyu Wu, Xunkai Li, Hongchao Qin

multimodal safety

Related stories