arXiv Machine Learning

Invariance before structure: two modes of emergent compression in representation learning

The paper introduces Emergent Adaptive Representation Learning (EARL), a framework that identifies two distinct modes of emergent compression in learned representations: state‑space compression, which reflects a representation’s growing predictive power over its own future, and parameter‑space compression, where networks with different micro‑level parameters converge to the same representation. Both modes are formalised using Fisher sensitivities and measured with ensemble‑level information‑theoretic metrics. Experiments on a recurrent reservoir with a trained bottleneck show that parameter‑space compression occurs early in training and is largely task‑independent, while state‑space compression emerges later, coinciding with the network’s learning of temporal task structure.

arXiv Machine Learning
Oct 28

FlowGenesis: Representation-Driven Self-Organization of Neural Architectures

FlowGenesis is a framework that lets neural networks self‑organize their architecture during training by adding or removing neurons based on a local statistic called Representation Flux. Neurons with high Flux are split to preserve function, while low‑Flux, low‑activity neurons are pruned. Experiments on image and text classification show that networks starting from different sizes converge to similar neuron counts and performance, and the method also improves continual learning by reducing parameters for new tasks without sacrificing accuracy.

By Maksim A Kazanskii
arXiv Machine Learning
Oct 22

Compression Through Time: An Interplay of Separability, Stationarity and Capacity

The paper investigates how information-theoretic and geometric compression unfolds over time in recurrent neural networks, specifically discretized LSTMs. It finds that compression is strongest at the final timesteps of many datasets and attributes this to three mechanisms: class separability, temporal stationarity of shared dynamics, and limited representational capacity. Through synthetic and real-world experiments, the authors demonstrate that these factors jointly determine the extent of temporal compression and its role as a regularizer when excess capacity exists, but can harm generalization when it consumes task-relevant memory.

By Nikolaus Kopp, Franz Pernkopf, Bernhard Geiger
arXiv AI
Jun 2

Rare Events, Real Signals: Functional Ensembles as Units of Computation in Deep Spiking Networks

arXiv:2606. 00073v1 Announce Type: cross Abstract: We investigate how internal representations emerge across hierarchical processing systems by introducing a neuroscience-inspired framework for analyzing deep spiking neural networks (SNN) through the lens of functional connectivity.

By Aditi Aravind, Konstantinos Ladakis, Mario Alexios Savaglio, Stelios M. Smirnakis, Maria Papadopouli
arXiv Computer Vision
Sep 3

Progression as Latent Drift: Generative Forecasting of Slow-Evolving Pathologies

The paper introduces Latent Drift, a generative forecasting framework that predicts slow-evolving neurodegenerative disease progression by learning changes in a compressed semantic representation rather than full-resolution anatomy. It addresses two failure modes—identity collapse and continuous interpolation trap—by removing pixel-level identity from the prediction target and applying Finite Scalar Quantization to suppress high-frequency nuisance fluctuations. Experiments on longitudinal 3D brain MRI demonstrate that Latent Drift outperforms diffusion and autoregressive transformer baselines in both generative fidelity and clinically relevant metrics.

By Yuxiang Feng, Juncheng Wang, Chao Xu, Wenlong Hou, Huihan Wang, Yijie Qian, Yang Liu, Baigui Sun, Yong Liu, Shujun Wang