The paper introduces the Multi-modal Knowledge Preserving Adapter (MKP-Adapter), an adapter-only approach that enables backward compatible training for multi-modal large language models without updating the backbone. It employs a multi-level preservation loss to maintain embedding geometry and a focal re-weighting strategy to focus on difficult samples. Experiments show strong backward compatibility across image, text, visual document, and video retrieval tasks with minimal latency overhead.
By Jaeseok Byun, Gukyeong Kwon, Han-Kai Hsu, Meher Gitika Karumuri, Zhikang Zhang, Hao Yang, Davide Modolo
arXiv:2607. 16305v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have achieved strong progress in multimodal understanding.
By Zeyu Xu, Xingzhong Hou, Pengkai Guo, Siling Lin, Xiao Xu, Menghua Zhai, Haoyu Chen, Yunke Zhang, Fei Huang
MoEMB introduces a mixture‑of‑experts (MoE) approach to scale universal multimodal embeddings (UME) without increasing the size of the output vector or relying on autoregressive decoding. By expanding encoder capacity along the expert axis, MoEMB achieves state‑of‑the‑art performance on MMEB‑V2 and MRMR benchmarks with only 3 B active parameters, outperforming TTE‑based methods that use more than four times as many active parameters and require significantly more compute. The paper also presents the first comprehensive study of adaptive computation for MoE‑based embeddings, exploring training‑time and inference‑time strategies to further improve efficiency for large‑scale retrieval and recommendation systems.
By Xuanming Cui, Shlok Kumar Mishra, Wentao Bao, Aashu Singh, Zihao Wang, Xiangjun Fan, Jun Xiao, Ser-Nam Lim, Jianpeng Cheng
arXiv:2607. 25266v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have enabled long-form video understanding at a scale that was not previously possible.
By Ghazal Kaviani, Ghassan AlRegib
arXiv:2611.05193v2 Announce Type: replace
Abstract: Composed Image Retrieval (CIR) aims to retrieve a target image given a reference image and a modification text. Although recent approaches leveragi...
By WonJun Moon, Seongchan Kim, Dahun Kim, Seungryong Kim
The paper introduces SRAIN, a framework that learns sample‑wise, rank‑aware interpolation weights for composed visual data retrieval. Instead of relying on complex multimodal large language models, SRAIN uses simple linear interpolation in embedding space, dynamically predicting query‑specific weights through batch‑wise rank‑aware estimation and a compact memory bank for hard negatives. This approach achieves state‑of‑the‑art performance on composed video retrieval and competitive results on composed image retrieval while significantly reducing query‑time latency.
By Boseung Jeong, Taegyu Park, Donghyeon Kwon, Hyunsouk Cho, Suha Kwak
The paper introduces PILOT, a framework that treats reasoning as a latent variable to enhance multimodal embeddings. By modeling prior and posterior reasoning modes with a shared large language model, PILOT uses relational supervision during training to guide exploration of retrieval‑relevant reasoning traces. On the MMEB‑V2 benchmark, PILOT outperforms baselines across all task groups, notably improving visual‑document retrieval scores by over six points.
By Tianyu Yang, Yijun Tian, Lars Benedikt Kaesberg, Jan Philip Wahle, Bela Gipp, Terry Ruas, Daniel Kurzawe
arXiv:2609.37225v1 Announce Type: cross
Abstract: Multimodal large language models (MLLMs) have shown strong potential for universal multimodal representation learning. However, existing methods eith...
By Zijing Cai, Yuzhe Wang, Jingxian Zhu, Fengbin Zhu, Richang Hong
arXiv:2511. 11421v2 Announce Type: replace-cross Abstract: Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge.
By Lan Li, Tao Hu, Da-Wei Zhou, Jia-Qi Yang, Han-Jia Ye, De-Chuan Zhan
arXiv:2606.13061v3 Announce Type: replace
Abstract: Reasoning-driven universal multimodal embedding has advanced rapidly by introducing Chain-of-Thought (CoT) reasoning into the embedding pipeline. D...
By Peixi Wu, Biao Yang, Feipeng Ma, Bosong Chai, Bo Lin, Wei Yuan, Fan Yang, Tingting Gao, Hebei Li, Xiaoyan Sun
CORE improves compositional reasoning in multimodal language models by distilling a cross‑attentive reranker’s fine‑grained judgments into the embedding model. It generates candidate lists across five compositional matching levels and trains with a Rank‑KL objective to replicate the reranker’s ranking. Experiments on COLA, SUGARCREPE++, and NEGBENCH show CORE‑RERANKER‑8B outperforms Jina‑Reranker by 10.7 points, while CORE‑EMBED‑8B achieves the best overall average among evaluated embeddings, with gains also transferring to the MCMR benchmark without harming COCO or Flickr30K retrieval.
By Tingyu Song, Mingxin Li, Yanzhao Zhang, Dingkun Long, Chu Liu, Pengjun Xie, Yilun Zhao, Shu Wu
arXiv:2606. 12809v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are trained on massive multimodal data, making data unlearning increasingly important as data owners may request the removal of specific content.
By He Li, Haoang Chi, Qizhou Wang, Yunxin Mao, Zhiheng Zhang, Jie Tan, Tongliang Liu, Wenjing Yang, Bo Han