← Back to all news
arXiv AI By Tianyu Yang, Yijun Tian, Lars Benedikt Kaesberg, Jan Philip Wahle, Bela Gipp, Terry Ruas, Daniel Kurzawe

Beyond Post-Hoc Rewards: Privileged Reasoning Exploration for Multimodal Embeddings

Read the original on arXiv AI →

The paper introduces PILOT, a framework that treats reasoning as a latent variable to enhance multimodal embeddings. By modeling prior and posterior reasoning modes with a shared large language model, PILOT uses relational supervision during training to guide exploration of retrieval‑relevant reasoning traces. On the MMEB‑V2 benchmark, PILOT outperforms baselines across all task groups, notably improving visual‑document retrieval scores by over six points.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computer Vision
Oct 12

VaME: Exploring Variational Latent Reasoning for Multimodal Embeddings

VaME (Variational Multimodal Embeddings) introduces a framework that treats latent reasoning as a learnable distribution over trajectories, enabling autoregressive exploration in latent space guided by answer reconstruction. By augmenting embedding-token readout with a latent-fused embedding and optimizing over stochastic variational trajectories via reinforcement learning with a Semantic Decoding Reward, VaME achieves superior performance on the MMEB-V2 benchmark and reasoning-intensive tasks like MRMR. The method also delivers at least a 4.25× inference speedup compared to deterministic latent autoregressive baselines.

By Peixi Wu, Mingzhou Jiang, Feipeng Ma, Biao Yang, Yunhao Zhou, Wei Yuan, Bosong Chai, Huizu Lin, Jie Chen, Zhangchi Hu, Fan Yang, Wenwu Ou, Hebei Li, Xiaoyan Sun
ragdiffusionreinforcement-learningmultimodalbenchmarks
More like this →
arXiv AI
Aug 20

UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval

UMER is a Unified Multimodal Embedding and Ranking framework that combines contrastive embeddings with Pair‑Aware Discriminative Reasoning to improve universal multimodal retrieval. It replaces item‑wise reflection with pair‑wise comparison of query–candidate pairs, enabling explicit identification of matching and discrepancy evidence. A mutual distillation strategy transfers reliable pairwise preferences between the embedding and ranking components, and UMER achieves state‑of‑the‑art performance on the MMEB‑V2 benchmark while supporting budget‑adjustable inference.

By Libiao Chen, Xiyang Liu, Yanheng Wei, Tao Wang, Zhenyu Tang
ragefficiencymultimodalbenchmarks
More like this →
arXiv AI
Sep 23

TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokens

arXiv:2605.16638v2 Announce Type: replace Abstract: Recent research has demonstrated that Universal Multimodal Embedding (UME) benefits significantly from Chain-of-Thought (CoT) reasoning. In this pa...

By Jianpeng Cheng, Xian Wu, Jiangfan Zhang, Wentao Bao, Chaitanya Ahuja, Shlok Kumar Mishra, Xuanming Cui, Hanchao Yu, Yang Gao, Fan Xia, Haixing Dai, Frankie Yuan, Zihao Wang, Xiaobing Chen, Qi Guo, Shaodan Zhai, Aashu Singh, Xiangjun Fan, Jun Xiao
llmsragmultimodalbenchmarks
More like this →
arXiv Computer Vision
Sep 1

Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

arXiv:2604.22280v4 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) have emerged as a promising foundation for universal multimodal embeddings. Recent studies have shown that...

By Peixi Wu, Ke Mei, Feipeng Ma, Bosong Chai, Zhibin Lan, Chenxi Zhao, Shannan Yan, Jie Chen, Zhangchi Hu, Yansong Peng, Bo Lin, Junjie Zhou, Dacheng Yin, Tianyi Wang, Fengyun Rao, Jing Lyu, Hebei Li, Xiaoyan Sun
llmsragreinforcement-learningmultimodalsafety
More like this →