arXiv AI
Jun 30
Multimodal Representation Alignment for Cross-modal Information Retrieval
arXiv:2506. 08774v2 Announce Type: replace-cross Abstract: Different machine learning models can represent the same underlying concept in different ways.
Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transport
arXiv:2608. 04234v1 Announce Type: cross Abstract: We study the problem of aligning data from multiple modalities into a shared representation space, focusing on settings where strong pretrained unimodal encoders are available but cross-modal paired data are scarce.
Related stories
arXiv Machine Learning
Jun 16
MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation
arXiv:2606. 16408v1 Announce Type: new Abstract: We introduce MUNI, an end-to-end multimodal latent diffusion framework for any-to-any generation that unifies subset-conditioned cross-modal generation and unconditional joint sampling through a shared stochastic latent.
arXiv AI
Jun 30
SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport
arXiv:2602. 23353v2 Announce Type: replace-cross Abstract: The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world.
arXiv Machine Learning
Jul 28
Distance-Matrix Wasserstein Statistics for Scalable Gromov--Wasserstein Learning
arXiv:2605. 14981v2 Announce Type: replace Abstract: Gromov--Wasserstein (GW) distances compare graphs, shapes, and point clouds through internal distances, without requiring a common coordinate system.
arXiv Machine Learning
Jun 4
KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models
arXiv:2606. 04180v1 Announce Type: new Abstract: Vision-language foundation models such as CLIP and SigLIP provide widely used representations for multimodal learning systems.
arXiv AI
Jun 30
Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation
arXiv:2606. 29464v1 Announce Type: cross Abstract: Vision-language dataset distillation (VLDD) compresses a large image-text paired dataset into a small set of synthetic pairs that can efficiently train contrastive vision-language models under strict data and compute budgets.
arXiv AI
Jul 21
Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
arXiv:2607. 17673v1 Announce Type: cross Abstract: Contrastive learning is increasingly moving toward settings with three or more modalities instead of image-text pairs.
arXiv Computer Vision
Sep 10
UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport
arXiv:2609.10224v1 Announce Type: new Abstract: Vision-language models such as CLIP embed images and text in a shared space, where modality-specific distributions often remain separated. Existing acc...
arXiv Machine Learning
Jun 9
Entropic Optimal Transport Eigenmaps for Nonlinear Alignment and Joint Embedding of High-Dimensional Datasets
arXiv:2407. 01718v2 Announce Type: replace-cross Abstract: Embedding high-dimensional data into a low-dimensional space is an indispensable component of data analysis.
arXiv Computer Vision
Aug 31
Relational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision
The study investigates whether transferring relational structure from human mental representations to deep neural networks (DNNs) can improve fine‑grained alignment between the two. Using unsupervised Gromov‑Wasserstein optimal transport, the authors show that fine‑tuning pre‑trained DNNs with Relational Knowledge Distillation (RKD) brings the networks close enough to human representations to align at the individual‑object level on a test set of concepts not seen during training. The improvement is driven mainly by a more human‑like global structure of category distances, while local nearest‑neighbor overlap remains largely unchanged.
arXiv AI
Aug 20
From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model
arXiv:2608. 18339v1 Announce Type: cross Abstract: Vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities yet remain sensitive to real-world distribution shifts during inference.
arXiv Machine Learning
Sep 25
GOMA: Toward Structure-Driven Multimodal Alignment from a Graph Signal Smoothing Perspective
GOMA (Graph-Optimized Multimodal Alignment) introduces a dual-embedding approach for multimodal retrieval, separating content embeddings supervised for paired identity from semantic embeddings trained with cross-modal pairs and observed relationships. The method fuses these embeddings, applies semantic agreement to weight graph edges, and uses restart graph propagation to reinforce the initial signal, enabling both single-modality and dual-attribute retrieval. Across six datasets and four tasks, GOMA outperforms 14 external methods on 14 primary metrics, with controlled experiments highlighting the impact of separate supervision, graph regularization, and semantic-guided propagation.