The paper introduces SCALAR, a query‑conditioned spherical centroid aggregator that assigns relevance‑based weights to each available modality before computing a spherical centroid. SCALAR supports arbitrary modality subsets, is trained with rank‑8 LoRA adapters on masked views, and achieves positive aggregation gains on four of five benchmarks, outperforming prior symmetric aggregators. With only 4.8 million trainable parameters, SCALAR attains the highest text‑to‑video R@1 on three benchmarks and surpasses the released GRAM checkpoint under test‑time modality dropout by 3.2 to 10.9 R@1.
By Ambuj Mehrish, Anindya Nag, Sebastiano Vascon
arXiv:2606. 14958v1 Announce Type: cross Abstract: We introduce the Massive Video Embedding Benchmark (MVEB), a 23-task benchmark for video embeddings spanning classification, zero-shot classification, clustering, pair classification, retrieval, and video-centric question answering.
By Adnan El Assadi, Roman Solomatin, Isaac Chung, Chenghao Xiao, Deep Shah, Manan Dey, Shriya Sudhakar, Zacharie Bugaud, Wissam Siblini, Ayush Sunil Munot, Yashwanth Devavarapu, Rakshitha Ireddi, Michelle Yang, M\'arton Kardos, Niklas Muennighoff, Kenneth Enevoldsen
arXiv:2609.37225v1 Announce Type: cross
Abstract: Multimodal large language models (MLLMs) have shown strong potential for universal multimodal representation learning. However, existing methods eith...
By Zijing Cai, Yuzhe Wang, Jingxian Zhu, Fengbin Zhu, Richang Hong
arXiv:2608. 16628v1 Announce Type: new Abstract: Modern Multimodal Retrieval-Augmented Generation (M-RAG) systems are fundamentally limited by the binary connectivity paradigm of traditional simple graphs, which fails to capture the intricate, high-order correlations among heterogeneous entities, such as the N-ary relationships between a visual chart, its scattered textual descriptions, and underlying numerical data.
By Shenao Chen, Yidan Xu, Xiangmin Han, Rundong Xue, Duanpo Wu, Yuhan Gao, Chenggang Yan, Yue Gao
TokenSwap introduces a method to benchmark and reduce the modality gap in multimodal large language models (MLLMs) by interleaving semantically aligned images with text tokens. By transforming text-only benchmarks like MMLU into image‑interleaved versions (TokenSwap‑Bench), the authors reveal a pervasive performance drop—averaging 19.6%—across 42 MLLMs, with reasoning models showing smaller gaps. Training with TokenSwap mitigates this gap while maintaining strong text‑only and vision‑language performance.
By Andong Hua, Colton Bishop, Igor Mordatch, Arian Hosseini, Jindong Gu, Aleksandra Faust, Rebecca Roelofs, Yao Qin
The paper investigates the problem of text dominance in multimodal large language models (MLLMs), where models rely heavily on textual input while underusing other modalities such as images, videos, audio, time‑series, and graphs. It introduces two metrics, the Modality Dominance Index (MDI) and the Attention Efficiency Index (AEI), to quantify this imbalance and finds that text dominance is significant across all tested modalities. The authors identify three causes—attention dilution from token redundancy, fusion architecture design, and task formulations favoring text—and propose a simple token compression technique that successfully reduces MDI in LLaVA‑7B from 10.23 to 0.86, offering a path toward more balanced multimodal models.
By Huyu Wu, Meng Tang, Xinhan Zheng, Haiyun Jiang
arXiv:2609.10224v1 Announce Type: new
Abstract: Vision-language models such as CLIP embed images and text in a shared space, where modality-specific distributions often remain separated. Existing acc...
By Zonglin Yang, Huilan Ma, Xudan Zheng, Yuejun Xie
arXiv:2608. 11343v1 Announce Type: new Abstract: Multimodal retrieval and classification across different types of media, spanning text, images,video and audio, has traditionally relied on dual-encoder models that align visual and textual representations through contrastive learning.
By Archan Dutta, Vyanktesh Kanungo
arXiv:2506. 08774v2 Announce Type: replace-cross Abstract: Different machine learning models can represent the same underlying concept in different ways.
By Fan Xu, Luis A. Leiva
The paper introduces HN-CLIP, a new objective for dense-caption retrieval that adapts similarity margins per negative example using the text encoder’s own geometry. By adding a detached caption‑similarity matrix to the negative logits, HN‑CLIP addresses the issue of near‑duplicate captions that cause premature loss saturation in InfoNCE training. Experiments on four benchmarks show that HN‑CLIP outperforms leading methods by 2.4–4.3 R@1, trains 2.4× faster than GOAL and 5.4× faster than StructXLIP, and achieves the best full‑data baseline with only 20% of the training data.
By Haoyue Liu, Ye Chen, Zhichao Wang, Xiaoying Tang
The paper explores training a large language model solely on textual video descriptions to perform zero‑shot video temporal localization, eliminating the need for paired video‑text data during training. Using a 134K‑example corpus and adapting Qwen3‑8B with LoRA on CLIP or LLM2CLIP embeddings, the authors evaluate on Charades‑STA and QV‑Highlights, achieving modest performance when directly replacing text with frame embeddings. They introduce Nearest Entity Replacement (NER), a training‑free mapping that represents each video frame by the mean of its nearest text‑entity embeddings, which significantly improves results and narrows the modality gap.
By Divyanshu Mishra, Sabine Sternig, Rakshith Shetty, Erhan Gundogdu
arXiv:2607. 25266v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have enabled long-form video understanding at a scale that was not previously possible.
By Ghazal Kaviani, Ghassan AlRegib