arXiv AI

Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping

arXiv:2508. 12466v2 Announce Type: replace-cross Abstract: Traditional multimodal learning approaches rely on alignment pre-training to bridge vision and language modalities, typically by projecting visual features into discrete text token spaces using large-scale image--text data.

arXiv AI

Jun 3

Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models

arXiv:2605. 18160v2 Announce Type: replace-cross Abstract: In recent years, multimodal large language models (MLLMs) have achieved remarkable progress, primarily attributed to effective paradigms for integrating visual and textual information.

By Xinpeng Dong, Min Zhang, Kairong Han, Xu Tan, Fei Wu, Kun Kuang

llms computer-vision multimodal benchmarks safety

arXiv AI

Jun 8

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

arXiv:2602. 07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions.

By Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan

llms rag fine-tuning multimodal safety

arXiv Machine Learning

Aug 12

MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment

arXiv:2608. 11167v1 Announce Type: cross Abstract: Existing Multimodal Large Language Models (MLLMs) predominantly rely on image-text pairs for modality alignment pretraining, mapping global image representations to long textual descriptions.

By Changhao Xiang, Shangyu Xing, Zhen Wu, Jianbing Zhang, Xinyu Dai

llms multimodal safety

arXiv AI

Jul 10

Reinforcing the Generation Order of Multimodal Masked Diffusion Models

arXiv:2607. 08056v1 Announce Type: cross Abstract: Diffusion Language Models (DLMs) have recently achieved substantial progress in natural language generation tasks.

By Yidong Ouyang, Zhe Wang, Sourav Bhabesh, Dmitriy Bespalov

llms diffusion multimodal benchmarks safety

arXiv Machine Learning

1d ago

FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens

arXiv:2506. 03096v2 Announce Type: replace-cross Abstract: Contrastive language-image pre-training aligns features of text-image pairs in a common latent space via distinct encoders for each modality.

By Christian Schlarmann, Francesco Croce, Nicolas Flammarion, Matthias Hein

llms rag diffusion multimodal

arXiv AI

Jul 14

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

arXiv:2604. 22823v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) rely on multimodal pre-training over diverse data sources, where different datasets often induce complementary cross-modal alignment capabilities.

By Zibo Shao, Baochen Xiong, Xiaoshan Yang, Yaguang Song, Qimeng Zhang, Haifeng Chen, Changsheng Xu

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jul 3

SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment

arXiv:2511. 01390v2 Announce Type: replace-cross Abstract: Fine-grained cross-modal alignment aims to establish precise local correspondences between vision and language, forming a cornerstone for visual question answering and related multimodal applications.

By Xinyu Mao, Junsi Li, Haoji Zhang, Yu Liang, Ming Sun

llms diffusion nlp multimodal safety

arXiv AI

Jun 30

Multimodal Representation Alignment for Cross-modal Information Retrieval

arXiv:2506. 08774v2 Announce Type: replace-cross Abstract: Different machine learning models can represent the same underlying concept in different ways.

By Fan Xu, Luis A. Leiva

llms rag multimodal benchmarks safety

arXiv AI

Jun 30

UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

arXiv:2603. 22282v2 Announce Type: replace-cross Abstract: We present UniMotion, to our knowledge the first unified framework for simultaneous understanding and generation of human motion, natural language, and RGB images within a single architecture.

By Ziyi Wang, Xinshun Wang, Shuang Chen, Yang Cong, Mengyuan Liu

llms nlp efficiency multimodal benchmarks safety

arXiv Machine Learning

Aug 4

Hierarchical Pre-Training of Vision Encoders with Large Language Model

arXiv:2604. 00086v2 Announce Type: replace-cross Abstract: The field of computer vision has experienced significant advancements through scalable vision encoders and multimodal pre-training frameworks.

By Eugene Lee, Ting-Yu Chang, Jui-Huang Tsai, Jiajie Diao, Chen-Yi Lee

llms rag computer-vision multimodal benchmarks safety

arXiv AI

Jun 17

Plug-and-Adapt: Multimodal Coreference Resolution at First Sight with a Pretrained Alignment Model

arXiv:2606. 17950v1 Announce Type: cross Abstract: Visual information helps resolve ambiguity in coreference resolution, leading to notable performance gains.

By Jinghan Wu, Jing Li, Ivor W. Tsang, Xuetao Zhang

multimodal benchmarks safety

arXiv AI

Jul 17

Stop Thinking, Start Looking: Efficient Post-Training for Multimodal Document Question Answering via Reasoning-Free Alignment

arXiv:2607. 14682v1 Announce Type: new Abstract: Efficient multimodal document question answering with explicit visual grounding, locating the precise document region that supports each answer remains an open challenge.

By Harikrishnan P M, Goutham Vignesh, Ganesh Parab, Saisubramaniam Gopalakrishnan, Vishal Vaddina, Varun V, Rohit Agrawal

nlp reinforcement-learning fine-tuning multimodal benchmarks safety

Related stories