ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
The Flow has not summarised this story yet — read it at arXiv AI.
arXiv:2506. 03096v2 Announce Type: replace-cross Abstract: Contrastive language-image pre-training aligns features of text-image pairs in a common latent space via distinct encoders for each modality.
arXiv:2606. 01207v1 Announce Type: cross Abstract: The choice between cross-attention and concatenation for multimodal fusion remains governed by practitioner intuition rather than principled understanding.
The paper introduces Modality Self-Play (MSP), a training framework that enables multimodal large language models to learn cross‑modal interactions without requiring paired non‑text data. MSP uses proxy‑mediated textual descriptions of unseen modalities as semantic bridges, allowing a model to generate and align these proxies with observed modalities. Experiments show that MSP achieves state‑of‑the‑art or competitive performance on several multimodal benchmarks and demonstrates effective zero‑shot modality composition through learned bridges and anchor‑supporting proxy descriptions.
arXiv:2508. 12466v2 Announce Type: replace-cross Abstract: Traditional multimodal learning approaches rely on alignment pre-training to bridge vision and language modalities, typically by projecting visual features into discrete text token spaces using large-scale image--text data.
arXiv:2604. 00086v2 Announce Type: replace-cross Abstract: The field of computer vision has experienced significant advancements through scalable vision encoders and multimodal pre-training frameworks.
Model Distribution-Aware Multimodal Dataset Distillation (MDA-DD) is a method that reduces the cost of vision‑language pre‑training by generating compact image‑text datasets. It uses a dynamic queue of models at different training stages to supervise distillation, introduces asymmetric dual projection for better alignment in visual and linguistic spaces, and employs an uncentered second‑order moment objective to preserve cross‑modal information. Experiments on Flickr30K and MS‑COCO show that with only 100 synthetic pairs, MDA-DD outperforms existing methods by significant margins on retrieval metrics, and it remains superior even when using 500 pairs.