Hugging Face Trending Papers

Model Distribution-Aware Multimodal Dataset Distillation

Read the original on Hugging Face Trending Papers →

Model Distribution-Aware Multimodal Dataset Distillation (MDA-DD) is a method that reduces the cost of vision‑language pre‑training by generating compact image‑text datasets. It uses a dynamic queue of models at different training stages to supervise distillation, introduces asymmetric dual projection for better alignment in visual and linguistic spaces, and employs an uncentered second‑order moment objective to preserve cross‑modal information. Experiments on Flickr30K and MS‑COCO show that with only 100 synthetic pairs, MDA-DD outperforms existing methods by significant margins on retrieval metrics, and it remains superior even when using 500 pairs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

arXiv Computation and Language
Oct 12

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

TokenSwap introduces a method to benchmark and reduce the modality gap in multimodal large language models (MLLMs) by interleaving semantically aligned images with text tokens. By transforming text-only benchmarks like MMLU into image‑interleaved versions (TokenSwap‑Bench), the authors reveal a pervasive performance drop—averaging 19.6%—across 42 MLLMs, with reasoning models showing smaller gaps. Training with TokenSwap mitigates this gap while maintaining strong text‑only and vision‑language performance.

By Andong Hua, Colton Bishop, Igor Mordatch, Arian Hosseini, Jindong Gu, Aleksandra Faust, Rebecca Roelofs, Yao Qin
arXiv AI
Jun 8

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

arXiv:2602. 07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions.

By Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan