arXiv AI

From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Model

arXiv:2508. 00955v3 Announce Type: replace-cross Abstract: Adapting generative Multimodal Large Language Models (MLLMs) into universal embedding models typically demands resource-intensive contrastive pre-training, while traditional hard negative mining methods suffer from severe false negative contamination.

arXiv AI

Jul 21

Learning to Detect Cross-Modal Negation: An Analysis of Latent Representations and an Attention-Based Solution

arXiv:2607. 17712v1 Announce Type: new Abstract: Detecting high-level semantic concepts like negation across modalities remains a challenge for current multimodal systems.

By Ali AbuSaleh, Leon Hammerla, Alexander Mehler

llms rag multimodal

arXiv Machine Learning

Jun 2

When Hard Negatives Hurt: Bridging the Generative-Discriminative Gap in Hard Negative Synthesis for Retrieval

arXiv:2606. 01304v1 Announce Type: new Abstract: Hard negative mining has become the dominant strategy for training retrievers, yet it faces intrinsic limitations: negatives are bounded by corpus availability, selected by retriever score rather than diagnostic value, and increasingly contaminated by false positives as the retriever improves.

By Zhicheng Zhang, Jiwei Tang, Kuicai Dong, Xiaopeng Li, Jieming Zhu, Jingyu Li, Qianhui Zhu, Fengyuan Lu, Wang Jiaheng, Gang Wang, Hai-Tao Zheng, Zhaocheng Du

llms

arXiv AI

6d ago

VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation

arXiv:2603. 17450v2 Announce Type: replace-cross Abstract: Sequential Recommendation (SR) in multimodal settings typically relies on small frozen pretrained encoders, which limits semantic capacity and prevents Collaborative Filtering (CF) signals from being fully integrated into item representations.

By Junyoung Kim, Woojoo Kim, Wonbin Kweon, Jaehyung Lim, Dongha Kim, Hwanjo Yu

llms rag fine-tuning multimodal

arXiv AI

Jun 8

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

arXiv:2602. 07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions.

By Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan

llms rag fine-tuning multimodal safety

Hugging Face Trending Papers

Jun 22

VideoLatent: Video-Language Learning via Latent Self-Forcing

Recent advancements in chain-of-thought (CoT) reasoning have shown promise in enhancing video understanding and reasoning capabilities of multimodal large language models (MLLMs). However, existing CoT-based MLLMs require labor-intensive CoT annotations and incur substantial training and inference overhead.

llms multimodal benchmarks safety

arXiv AI

Jul 16

Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping

arXiv:2508. 12466v2 Announce Type: replace-cross Abstract: Traditional multimodal learning approaches rely on alignment pre-training to bridge vision and language modalities, typically by projecting visual features into discrete text token spaces using large-scale image--text data.

By Xuhui Zhan, Tyler Derr

llms rag multimodal benchmarks safety

arXiv AI

Jul 28

Controlling Embedding Spaces with Text-Conditioned Transformations

arXiv:2607. 22919v1 Announce Type: cross Abstract: Multimodal embedding spaces in models like CLIP enable powerful capabilities such as semantic similarity retrieval and cross-modal zero-shot classification.

By Joseph Fioresi, Fabian Caba Heilbron, Pankaj Nathani, Mubarak Shah, Kushal Kafle

rag diffusion multimodal benchmarks

arXiv AI

Jun 19

ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval

arXiv:2606. 20280v1 Announce Type: cross Abstract: Leveraging Multimodal Large Language Models (MLLMs) via contrastive learning has become a mainstream paradigm for improving the performance of Universal Multimodal Retrieval (UMR).

By Yuhan Liu, Pei Fu, Hang Li, Yukun Qi, Chao Jiang, Jingwen Fu, Zhen Liu, Bin Qin, Zhenbo Luo, Jian Luan, Jingmin Xin

llms reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 2

Reconstructing Content via Collaborative Attention to Improve Multimodal Embedding Quality

arXiv:2603. 01471v2 Announce Type: replace-cross Abstract: Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification.

By Jiahan Chen, Da Li, Hengran Zhang, Yinqiong Cai, Lixin Su, Jiafeng Guo, Daiting Shi, Dawei Yin, Keping Bi

llms rag multimodal

arXiv AI

Jun 30

UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

arXiv:2603. 22282v2 Announce Type: replace-cross Abstract: We present UniMotion, to our knowledge the first unified framework for simultaneous understanding and generation of human motion, natural language, and RGB images within a single architecture.

By Ziyi Wang, Xinshun Wang, Shuang Chen, Yang Cong, Mengyuan Liu

llms nlp efficiency multimodal benchmarks safety

arXiv AI

Jul 21

LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models

arXiv:2607. 16305v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have achieved strong progress in multimodal understanding.

By Zeyu Xu, Xingzhong Hou, Pengkai Guo, Siling Lin, Xiao Xu, Menghua Zhai, Haoyu Chen, Yunke Zhang, Fei Huang

llms rag multimodal benchmarks

arXiv Machine Learning

Jun 2

Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs

arXiv:2606. 01710v1 Announce Type: cross Abstract: Vision-Language models (VLMs), such as CLIP, achieve powerful zero-shot classification.

By Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani

llms rag fine-tuning multimodal benchmarks safety

Related stories