← Back to all news
arXiv AI By Randall Balestriero, Yann LeCun

Proving and Improving the Platonic Representation Hypothesis with JEPAs

Read the original on arXiv AI →

The paper proves the Platonic Representation Hypothesis by showing that Joint-Embedding Predictive Architectures (JEPAs) trained on different modalities converge to the same representation up to a rotation when their positive pairs are good views of the same scene. It demonstrates that next‑token prediction is a JEPA with bad views, explaining why language models only align with vision through large scale. Experiments on millions of captions and web images confirm a scaling law of alignment, revealing that good views require far fewer samples than next‑token prediction and that a 109M‑parameter text JEPA outperforms larger language models in aligning with image encoders, even enabling a simple rotation to produce an image captioner with high COCO CIDEr scores.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 8

Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

The paper proposes that independently trained multimodal models often share a common geometric structure, allowing cross‑modal alignment without paired data. Using a Wasserstein Procrustes approach with a coarse geometric initialization, the authors align two disjoint embedding sets by estimating a single orthogonal map, achieving consistent alignment across datasets and modalities. While paired examples can further improve performance, the method already outperforms existing techniques in the few‑pair regime and enables text‑to‑image generation without paired data.

By Dominik Schnaus, Thomas Dag\`es, Daniel Cremers, Xi Wang, Phillip Isola
ragdiffusionmultimodalsafety
More like this →
arXiv Machine Learning
Oct 15

From Plato to Aristotle: Toward a General Theory of Representation Hypotheses

The paper introduces the Lattice Representation Hypothesis (LRH) to explain why different alignment criteria—such as neighbor overlap, spanning trees, and distances—yield varying views on model representation convergence. It shows that the Platonic Representation Hypothesis (PRH) reflects agreement at the top of a lattice, while evidence comes from criteria near the bottom, and that the Aristotelian Representation Hypothesis (ARH) limits convergence to local neighborhoods due to calibration effects. Experiments on numerous pretrained vision and language models, along with theoretical analyses of linear autoencoders and contrastive Gibbs models, support the LRH’s claim that training procedures and objectives determine which criteria are reliably met across runs.

By Dai Shi, Andi Han, Xiaoyu Li, Luke Thompson, Jos\'e Miguel Hern\'andez-Lobato
llmssafety
More like this →