arXiv AI
Oct 31

Proving and Improving the Platonic Representation Hypothesis with JEPAs

The paper proves the Platonic Representation Hypothesis by showing that Joint-Embedding Predictive Architectures (JEPAs) trained on different modalities converge to the same representation up to a rotation when their positive pairs are good views of the same scene. It demonstrates that next‑token prediction is a JEPA with bad views, explaining why language models only align with vision through large scale. Experiments on millions of captions and web images confirm a scaling law of alignment, revealing that good views require far fewer samples than next‑token prediction and that a 109M‑parameter text JEPA outperforms larger language models in aligning with image encoders, even enabling a simple rotation to produce an image captioner with high COCO CIDEr scores.

By Randall Balestriero, Yann LeCun
arXiv Machine Learning
Oct 15

From Plato to Aristotle: Toward a General Theory of Representation Hypotheses

The paper introduces the Lattice Representation Hypothesis (LRH) to explain why different alignment criteria—such as neighbor overlap, spanning trees, and distances—yield varying views on model representation convergence. It shows that the Platonic Representation Hypothesis (PRH) reflects agreement at the top of a lattice, while evidence comes from criteria near the bottom, and that the Aristotelian Representation Hypothesis (ARH) limits convergence to local neighborhoods due to calibration effects. Experiments on numerous pretrained vision and language models, along with theoretical analyses of linear autoencoders and contrastive Gibbs models, support the LRH’s claim that training procedures and objectives determine which criteria are reliably met across runs.

By Dai Shi, Andi Han, Xiaoyu Li, Luke Thompson, Jos\'e Miguel Hern\'andez-Lobato