arXiv Statistics ML

The Information Geometry of Representation Formation

arXiv AI
Oct 16

Learning Intrinsic Dynamics in Latent World Models

The paper investigates how encouraging action predictability in latent world models can preserve intrinsic dynamics, which are crucial for planning. By deriving a theoretical lower bound and conducting empirical tests, the authors show that action predictability prevents the encoder from merging distinct action outcomes, even when visual appearance varies. Using a method called Concat IDM, they achieve a substantial increase in planning success on the Push‑T benchmark, from 9.1% to 77.2%, demonstrating the practical benefit of this inductive bias.

By Jiamin Shi, Yuanhong Zhang, Zhiyang Liu, Jiawei Sun, Jingwen Fu
arXiv AI
Jun 17

Dimensionality Controls When Modularity Helps in Continual Learning

arXiv:2606. 17889v1 Announce Type: cross Abstract: Compositional learning systems must balance plasticity, the ability to acquire new knowledge, with stability, the preservation of previously learned components, especially when tasks share structure and risk interference.

By Kathrin Korte, Christian Medeiros Adriano, Joachim Winther Pedersen, Eleni Nisioti, Sebastian Risi
arXiv Machine Learning
Nov 6

Invariance before structure: two modes of emergent compression in representation learning

The paper introduces Emergent Adaptive Representation Learning (EARL), a framework that identifies two distinct modes of emergent compression in learned representations: state‑space compression, which reflects a representation’s growing predictive power over its own future, and parameter‑space compression, where networks with different micro‑level parameters converge to the same representation. Both modes are formalised using Fisher sensitivities and measured with ensemble‑level information‑theoretic metrics. Experiments on a recurrent reservoir with a trained bottleneck show that parameter‑space compression occurs early in training and is largely task‑independent, while state‑space compression emerges later, coinciding with the network’s learning of temporal task structure.

By Hardik Rajpal, Dan Goodman
arXiv Machine Learning
Nov 5

Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis

The paper challenges the assumption that improved performance in deep learning automatically leads to better internal representations. By comparing networks evolved through open‑ended search with those trained by stochastic gradient descent on a simple image‑generation task, the authors find that both produce identical outputs but differ markedly in their internal structure. The SGD‑trained models exhibit a fractured entangled representation (FER), whereas the evolved models approach a unified factored representation (UFR).

By Akarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanley