arXiv Machine Learning By Hardik Rajpal, Dan Goodman

Invariance before structure: two modes of emergent compression in representation learning

Read the original on arXiv Machine Learning →

The paper introduces Emergent Adaptive Representation Learning (EARL), a framework that identifies two distinct modes of emergent compression in learned representations: state‑space compression, which reflects a representation’s growing predictive power over its own future, and parameter‑space compression, where networks with different micro‑level parameters converge to the same representation. Both modes are formalised using Fisher sensitivities and measured with ensemble‑level information‑theoretic metrics. Experiments on a recurrent reservoir with a trained bottleneck show that parameter‑space compression occurs early in training and is largely task‑independent, while state‑space compression emerges later, coinciding with the network’s learning of temporal task structure.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning
Oct 28

FlowGenesis: Representation-Driven Self-Organization of Neural Architectures

FlowGenesis is a framework that lets neural networks self‑organize their architecture during training by adding or removing neurons based on a local statistic called Representation Flux. Neurons with high Flux are split to preserve function, while low‑Flux, low‑activity neurons are pruned. Experiments on image and text classification show that networks starting from different sizes converge to similar neuron counts and performance, and the method also improves continual learning by reducing parameters for new tasks without sacrificing accuracy.

By Maksim A Kazanskii
arXiv Machine Learning
Oct 22

Compression Through Time: An Interplay of Separability, Stationarity and Capacity

The paper investigates how information-theoretic and geometric compression unfolds over time in recurrent neural networks, specifically discretized LSTMs. It finds that compression is strongest at the final timesteps of many datasets and attributes this to three mechanisms: class separability, temporal stationarity of shared dynamics, and limited representational capacity. Through synthetic and real-world experiments, the authors demonstrate that these factors jointly determine the extent of temporal compression and its role as a regularizer when excess capacity exists, but can harm generalization when it consumes task-relevant memory.

By Nikolaus Kopp, Franz Pernkopf, Bernhard Geiger