← Back to all news
arXiv Machine Learning By Dominik Schnaus, Thomas Dag\`es, Daniel Cremers, Xi Wang, Phillip Isola

Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

Read the original on arXiv Machine Learning →

The paper proposes that independently trained multimodal models often share a common geometric structure, allowing cross‑modal alignment without paired data. Using a Wasserstein Procrustes approach with a coarse geometric initialization, the authors align two disjoint embedding sets by estimating a single orthogonal map, achieving consistent alignment across datasets and modalities. While paired examples can further improve performance, the method already outperforms existing techniques in the few‑pair regime and enables text‑to‑image generation without paired data.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jun 8

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

arXiv:2602. 07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions.

By Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan
llmsragfine-tuningmultimodalsafety
More like this →
arXiv AI
Oct 16

MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

The paper introduces Modality Self-Play (MSP), a training framework that enables multimodal large language models to learn cross‑modal interactions without requiring paired non‑text data. MSP uses proxy‑mediated textual descriptions of unseen modalities as semantic bridges, allowing a model to generate and align these proxies with observed modalities. Experiments show that MSP achieves state‑of‑the‑art or competitive performance on several multimodal benchmarks and demonstrates effective zero‑shot modality composition through learned bridges and anchor‑supporting proxy descriptions.

By Robert Moseley, Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian, Salman Asif
llmsreinforcement-learningmultimodalbenchmarkssafety
More like this →