arXiv AI
When Fusion Order Matters: Insights on Progressive Multimodal Representation Learning for Cross-modal Retrieval
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computer Vision
Sep 15
Query-Conditioned Spherical Centroid Aggregation for Multimodal Retrieval
The paper introduces SCALAR, a query‑conditioned spherical centroid aggregator that assigns relevance‑based weights to each available modality before computing a spherical centroid. SCALAR supports arbitrary modality subsets, is trained with rank‑8 LoRA adapters on masked views, and achieves positive aggregation gains on four of five benchmarks, outperforming prior symmetric aggregators. With only 4.8 million trainable parameters, SCALAR attains the highest text‑to‑video R@1 on three benchmarks and surpasses the released GRAM checkpoint under test‑time modality dropout by 3.2 to 10.9 R@1.
arXiv Machine Learning
Jun 3
Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals
arXiv:2606. 02679v1 Announce Type: new Abstract: Multimodal systems often benefit from combining information across language, sound, and visual streams, but this benefit is not guaranteed.
arXiv Computation and Language
Oct 26
A Broader Look into MLLM's Retrieval Heads
The paper investigates how retrieval heads—sparse attention heads that transfer information from context to output—behave when multimodal large language models (MLLMs) are extended to include audio-visual inputs. Across 18 language, vision‑language, and omni‑modal checkpoints, the authors find that retrieval heads remain sparse and largely shared across modalities, with less than 2 % of heads being common to all modalities; ablating these shared heads degrades tasks such as document understanding, spatial perception, and visual mathematics. By training retrieval heads with a ranking objective, they demonstrate a 6.03‑point improvement in Recall@1 on the MMDocIR benchmark for Qwen3‑VL‑8B and improved document retrieval across three model families.
arXiv AI
Aug 14
Generative Universal Multimodal Retrieval with Dual-role Identifiers
arXiv:2608. 12987v1 Announce Type: cross Abstract: Generative information retrieval (GIR) has emerged as a compelling alternative to the conventional index-retrieve-then-rank retrieval pipeline by training a generator to produce the identifiers of relevant items directly.
arXiv AI
Jun 17
Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusion
arXiv:2603. 22372v2 Announce Type: replace-cross Abstract: Recent advances in multimodal learning have motivated the integration of auxiliary modalities such as text or vision into time series (TS) forecasting.
arXiv AI
Jun 29
HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering
arXiv:2603. 18558v2 Announce Type: replace-cross Abstract: Long-form video question answering requires reasoning over extended temporal contexts, making frame selection a critical bottleneck for multi-modal large language models (MLLMs) bound by finite context windows.