← Back to all news
arXiv Machine Learning By Nikolaus Kopp, Franz Pernkopf, Bernhard Geiger

Compression Through Time: An Interplay of Separability, Stationarity and Capacity

Read the original on arXiv Machine Learning →

The paper investigates how information-theoretic and geometric compression unfolds over time in recurrent neural networks, specifically discretized LSTMs. It finds that compression is strongest at the final timesteps of many datasets and attributes this to three mechanisms: class separability, temporal stationarity of shared dynamics, and limited representational capacity. Through synthetic and real-world experiments, the authors demonstrate that these factors jointly determine the extent of temporal compression and its role as a regularizer when excess capacity exists, but can harm generalization when it consumes task-relevant memory.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Sep 10

Memory in Deep Time-Series Models

arXiv:2609.06006v1 Announce Type: cross Abstract: Deep learning for time series has progressed through successive architectural paradigms, from recurrent networks and transformers to structured state...

By Minh Hoang Nguyen, Huu Hiep Nguyen, Manh Nguyen, Van Dai Do, Dung Nguyen, Hung Le
llmsragagents
More like this →
arXiv Machine Learning
Aug 19

Dynamic Compression in Recurrent Networks

Dynamic Compression in Recurrent Networks proposes a method that lets recurrent models revisit and revise their fixed-size state through additional updates, rather than compressing all information in a single causal pass. This approach allows the model to retain lower-fidelity history and refine only the relevant parts when needed, reducing the required state size for accurate task reuse. Experiments show that dynamic compression lowers the recurrent state needed and scales better as the number of stored functions increases.

By Jyothish Pari, Ryan Bahlous-Boldi, Pulkit Agrawal
More like this →
arXiv AI
Sep 25

ELiSe: Efficient Learning of Sequences in Structured Recurrent Networks

The paper introduces ELiSe, a model that leverages cortical network scaffolds and dendritic compartments to learn complex non‑Markovian spatio‑temporal patterns using only local, always‑on, phase‑free synaptic plasticity. It demonstrates the model’s ability to acquire and replay intricate sequences, exemplified by a birdsong learning mock‑up, and shows robustness to external disturbances and flexibility in parameter settings.

By Laura Kriener, Kristin V\"olk, Ben von H\"unerbein, Federico Benitez, Walter Senn, Mihai A. Petrovici
More like this →
Hugging Face Trending Papers
Aug 18

Dynamic Compression in Recurrent Networks

Dynamic Compression in Recurrent Networks proposes a method for recurrent models to selectively revisit and update past tokens, rather than compressing all history in a single causal pass. By allowing the model to refine its fixed-size state only when needed, it can maintain lower-fidelity information in the raw sequence and revisit it later. Experiments show that this selective re-scanning reduces the recurrent state needed for accurate task reuse and scales better as the number of stored functions increases.

More like this →