arXiv AI By Zijing Cai, Yuzhe Wang, Jingxian Zhu, Fengbin Zhu, Richang Hong

ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression

Read the original on arXiv AI →

The Flow has not summarised this story yet — read it at arXiv AI.

arXiv Computation and Language
Sep 23

CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding

CausalEmbed is an auto‑regressive method for generating compact multi‑vector embeddings in visual document retrieval. By using iterative margin loss during contrastive training, it reduces the number of visual tokens needed by 30‑155× while keeping performance competitive across different backbones and benchmarks. The approach offers efficient training, scalable test‑time performance, and a flexible scaling strategy for multi‑vector representations.

By Jiahao Huo, Yu Huang, Yibo Yan, Ye Pan, Kening Zheng, Wei-Chieh Huang, Yi Cao, Mingdong Ou, Philip S. Yu, Xuming Hu
arXiv Computation and Language
Oct 12

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

TokenSwap introduces a method to benchmark and reduce the modality gap in multimodal large language models (MLLMs) by interleaving semantically aligned images with text tokens. By transforming text-only benchmarks like MMLU into image‑interleaved versions (TokenSwap‑Bench), the authors reveal a pervasive performance drop—averaging 19.6%—across 42 MLLMs, with reasoning models showing smaller gaps. Training with TokenSwap mitigates this gap while maintaining strong text‑only and vision‑language performance.

By Andong Hua, Colton Bishop, Igor Mordatch, Arian Hosseini, Jindong Gu, Aleksandra Faust, Rebecca Roelofs, Yao Qin
arXiv Machine Learning
Jun 3

Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

arXiv:2603. 01471v3 Announce Type: replace-cross Abstract: Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification.

By Jiahan Chen, Da Li, Hengran Zhang, Yinqiong Cai, Lixin Su, Jiafeng Guo, Daiting Shi, Dawei Yin, Keping Bi
arXiv Machine Learning
Jun 2

Reconstructing Content via Collaborative Attention to Improve Multimodal Embedding Quality

arXiv:2603. 01471v2 Announce Type: replace-cross Abstract: Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification.

By Jiahan Chen, Da Li, Hengran Zhang, Yinqiong Cai, Lixin Su, Jiafeng Guo, Daiting Shi, Dawei Yin, Keping Bi
arXiv AI
Oct 31

PACE: Pareto-Adaptive Compression for Efficient Native MLLMs

The paper introduces PACE, a Pareto‑adaptive compression framework designed for native multimodal large language models (MLLMs) that directly tokenize pixel patches without external vision encoders. By analyzing token discriminability and heterogeneous redundancy across spatial dimensions, PACE decomposes attention into height‑ and width‑specific saliency maps and applies a Pareto‑optimization strategy to allocate token budgets, achieving near‑lossless performance with only 25‑50% of visual tokens and more than 2× FLOP savings. Experiments demonstrate that PACE not only preserves but sometimes improves benchmark results compared to existing ViT‑based compression methods.

By Lei Tian, Chunzheng Zhu, Kai Wu, Yifan Zhang, Haihua Yang