arXiv AI By Hanpeng Liu, Zidan Wang, Shuoxi Zhang, Zonglin Zhao, Zihao Bo, Rinyoichi Takezoe, Kaiwen Long, Yaqian Li, Kun He

ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining

Read the original on arXiv AI →

The Flow has not summarised this story yet — read it at arXiv AI.

arXiv AI
Oct 16

MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

The paper introduces Modality Self-Play (MSP), a training framework that enables multimodal large language models to learn cross‑modal interactions without requiring paired non‑text data. MSP uses proxy‑mediated textual descriptions of unseen modalities as semantic bridges, allowing a model to generate and align these proxies with observed modalities. Experiments show that MSP achieves state‑of‑the‑art or competitive performance on several multimodal benchmarks and demonstrates effective zero‑shot modality composition through learned bridges and anchor‑supporting proxy descriptions.

By Robert Moseley, Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian, Salman Asif
Hugging Face Trending Papers
5d ago

Model Distribution-Aware Multimodal Dataset Distillation

Model Distribution-Aware Multimodal Dataset Distillation (MDA-DD) is a method that reduces the cost of vision‑language pre‑training by generating compact image‑text datasets. It uses a dynamic queue of models at different training stages to supervise distillation, introduces asymmetric dual projection for better alignment in visual and linguistic spaces, and employs an uncentered second‑order moment objective to preserve cross‑modal information. Experiments on Flickr30K and MS‑COCO show that with only 100 synthetic pairs, MDA-DD outperforms existing methods by significant margins on retrieval metrics, and it remains superior even when using 500 pairs.