arXiv AI

Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances

arXiv:2405. 12775v2 Announce Type: replace-cross Abstract: Discovering the semantics of multimodal utterances is essential for understanding human language and enhancing human-machine interactions.

arXiv AI

Jun 2

NILC: Discovering New Intents with LLM-assisted Clustering

arXiv:2511. 05913v2 Announce Type: replace-cross Abstract: New intent discovery (NID) seeks to recognize both new and known intents from unlabeled user utterances, which finds prevalent use in practical dialogue systems.

By Hongtao Wang, Renchi Yang, Wenqing Lin

llms rag benchmarks

arXiv AI

Jun 9

Bridging Traditional Explainability Methods and Multimodal Multilingual Models: An XAI-Based Analysis

arXiv:2606. 07533v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) effectively integrate text and audio to interpret context in complex interactive dialogues.

By Pawe{\l} Pozorski, Jakub Muszy\'nski, Maria Ganzha

llms nlp multimodal safety

arXiv Machine Learning

Jul 21

MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning

arXiv:2607. 16789v1 Announce Type: new Abstract: Real-world perception and decision making are inherently multimodal, integrating complementary signals across modalities.

By Sana Tonekaboni, Viktoria Schuster, Caroline Uhler

fine-tuning multimodal benchmarks safety

arXiv AI

Jun 8

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

arXiv:2602. 07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions.

By Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan

llms rag fine-tuning multimodal safety

arXiv Machine Learning

Jul 2

Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning

arXiv:2505. 19614v2 Announce Type: replace Abstract: Multimodal learning has seen remarkable progress, particularly with large-scale pre-training across various modalities.

By Sanghyuk Chun, Olga Russakovsky

multimodal benchmarks safety

arXiv AI

Aug 5

From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Model

arXiv:2508. 00955v3 Announce Type: replace-cross Abstract: Adapting generative Multimodal Large Language Models (MLLMs) into universal embedding models typically demands resource-intensive contrastive pre-training, while traditional hard negative mining methods suffer from severe false negative contamination.

By Yeong-Joon Ju, Seong-Whan Lee

llms rag fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 16

Unsupervised Learning for Missing Modalities in Multimodal Learning

arXiv:2606. 15743v1 Announce Type: new Abstract: This paper addresses the missing-modality challenge in multi-modal learning by introducing Unsupervised Learning for Missing Modalities in Multi-Modal Learning (UL4M4), a flexible framework that imputes missing feature embeddings in a task-independent manner before supervised prediction.

By Hassan Ismkhan, Hamid Bouchahcia

rag multimodal benchmarks

arXiv Machine Learning

Jun 3

Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

arXiv:2603. 01471v3 Announce Type: replace-cross Abstract: Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification.

By Jiahan Chen, Da Li, Hengran Zhang, Yinqiong Cai, Lixin Su, Jiafeng Guo, Daiting Shi, Dawei Yin, Keping Bi

llms rag multimodal

arXiv Machine Learning

Jun 2

Reconstructing Content via Collaborative Attention to Improve Multimodal Embedding Quality

arXiv:2603. 01471v2 Announce Type: replace-cross Abstract: Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification.

By Jiahan Chen, Da Li, Hengran Zhang, Yinqiong Cai, Lixin Su, Jiafeng Guo, Daiting Shi, Dawei Yin, Keping Bi

llms rag multimodal

arXiv AI

Aug 11

MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks

arXiv:2507. 19634v4 Announce Type: replace-cross Abstract: Recent advances in large language models have laid the foundation for multimodal LLMs (MLLMs), which unify text, speech, and vision within a single framework.

By Sara Papi, Maike Z\"ufle, Marco Gaido, Beatrice Savoldi, Danni Liu, Ioannis Douros, Luisa Bentivogli, Jan Niehues

llms nlp multimodal benchmarks

arXiv AI

Jul 9

Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts

arXiv:2607. 06611v1 Announce Type: cross Abstract: Automatically recognizing the sentiment, positive or negative, from speech is a challenging task, requiring both the analysis of vocal inflections and the interpretation of uttered words.

By Andrei-George Durdun, Victor Constantinescu, Radu Tudor Ionescu

llms nlp efficiency multimodal benchmarks

arXiv AI

Jun 29

HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering

arXiv:2603. 18558v2 Announce Type: replace-cross Abstract: Long-form video question answering requires reasoning over extended temporal contexts, making frame selection a critical bottleneck for multi-modal large language models (MLLMs) bound by finite context windows.

By Dan Ben-Ami, Gabriele Serussi, Kobi Cohen, Chaim Baskin

llms rag computer-vision nlp multimodal benchmarks

Related stories