The paper introduces Lens, a training‑free framework that aligns multimodal representations with the semantic perspective required by downstream tasks. Lens uses a task‑specific readout phrase to anchor the perspective and then aggregates token states after the full input, ensuring the extracted representation reflects task‑conditioned evidence integration rather than generic salient content. The method achieves a Precision@1 of 63.9 across 36 MMEB datasets, outperforming the nearest training‑free baseline by 10.2 points.
arXiv:2607. 14682v1 Announce Type: new Abstract: Efficient multimodal document question answering with explicit visual grounding, locating the precise document region that supports each answer remains an open challenge.
arXiv:2606. 17950v1 Announce Type: cross Abstract: Visual information helps resolve ambiguity in coreference resolution, leading to notable performance gains.
arXiv:2610.08341v1 Announce Type: cross Abstract: Recent training-free pruning approaches for Multimodal Large Language Models (MLLMs) effectively cut computational overhead by exploiting visual redu...
VIVAS is a new Vision‑Language Model pre‑training framework that addresses the lack of fine‑grained visual perception in existing VLMs. It introduces a unified token space and a dense‑structural‑semantic vision tokenizer that expands the textual vocabulary with visual tokens, enabling vision‑language unified autoregressive supervision over both visual details and linguistic content. Trained on 12.4 T tokens, VIVAS achieves state‑of‑the‑art results on 7 tasks and 39 multimodal benchmarks.
arXiv:2503. 06211v3 Announce Type: replace-cross Abstract: Text-pretrained language models (LMs) encode rich world knowledge, but adapting them to process and generate perceptual modalities such as audio and images while effectively leveraging that knowledge remains challenging.
TokenSwap introduces a method to benchmark and reduce the modality gap in multimodal large language models (MLLMs) by interleaving semantically aligned images with text tokens. By transforming text-only benchmarks like MMLU into image‑interleaved versions (TokenSwap‑Bench), the authors reveal a pervasive performance drop—averaging 19.6%—across 42 MLLMs, with reasoning models showing smaller gaps. Training with TokenSwap mitigates this gap while maintaining strong text‑only and vision‑language performance.
arXiv:2604.22280v4 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) have emerged as a promising foundation for universal multimodal embeddings. Recent studies have shown that...
The paper introduces a framework that aligns multimodal embeddings with language by using captioning functions on interpretable auxiliary data, a soft contrastive loss, and a rank-based concept retrieval task. Applied to Earth observation embeddings, the method improves species distribution modelling, achieving state‑of‑the‑art performance on most metrics and better concept retrieval than prior vision‑language models. The aligned concepts enable recovery of species‑habitat associations and allow domain experts to interpret model predictions in natural language.
The paper introduces PILOT, a framework that treats reasoning as a latent variable to enhance multimodal embeddings. By modeling prior and posterior reasoning modes with a shared large language model, PILOT uses relational supervision during training to guide exploration of retrieval‑relevant reasoning traces. On the MMEB‑V2 benchmark, PILOT outperforms baselines across all task groups, notably improving visual‑document retrieval scores by over six points.
arXiv:2611.05193v2 Announce Type: replace Abstract: Composed Image Retrieval (CIR) aims to retrieve a target image given a reference image and a modification text. Although recent approaches leveragi...
arXiv:2608. 03450v1 Announce Type: cross Abstract: Reasoning in Multimodal Large Language Models (MLLMs) requires both fine-grained visual perception and rigorous logical deduction.
arXiv:2605. 18160v2 Announce Type: replace-cross Abstract: In recent years, multimodal large language models (MLLMs) have achieved remarkable progress, primarily attributed to effective paradigms for integrating visual and textual information.