The paper investigates how information-theoretic and geometric compression unfolds over time in recurrent neural networks, specifically discretized LSTMs. It finds that compression is strongest at the final timesteps of many datasets and attributes this to three mechanisms: class separability, temporal stationarity of shared dynamics, and limited representational capacity. Through synthetic and real-world experiments, the authors demonstrate that these factors jointly determine the extent of temporal compression and its role as a regularizer when excess capacity exists, but can harm generalization when it consumes task-relevant memory.
arXiv:2609.06006v1 Announce Type: cross Abstract: Deep learning for time series has progressed through successive architectural paradigms, from recurrent networks and transformers to structured state...
Dynamic Compression in Recurrent Networks proposes a method that lets recurrent models revisit and revise their fixed-size state through additional updates, rather than compressing all information in a single causal pass. This approach allows the model to retain lower-fidelity history and refine only the relevant parts when needed, reducing the required state size for accurate task reuse. Experiments show that dynamic compression lowers the recurrent state needed and scales better as the number of stored functions increases.
The paper introduces ELiSe, a model that leverages cortical network scaffolds and dendritic compartments to learn complex non‑Markovian spatio‑temporal patterns using only local, always‑on, phase‑free synaptic plasticity. It demonstrates the model’s ability to acquire and replay intricate sequences, exemplified by a birdsong learning mock‑up, and shows robustness to external disturbances and flexibility in parameter settings.
arXiv:2607. 13432v1 Announce Type: new Abstract: Plasticity -- a neural network's ability to adapt to new tasks -- is critical for continual and transfer learning.
Dynamic Compression in Recurrent Networks proposes a method for recurrent models to selectively revisit and update past tokens, rather than compressing all history in a single causal pass. By allowing the model to refine its fixed-size state only when needed, it can maintain lower-fidelity information in the raw sequence and revisit it later. Experiments show that this selective re-scanning reduces the recurrent state needed for accurate task reuse and scales better as the number of stored functions increases.
arXiv:2609.16540v1 Announce Type: cross Abstract: State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear comput...
arXiv:2607. 10430v1 Announce Type: cross Abstract: Dimensionality reduction has proven powerful for identifying neural manifolds, which are low-dimensional structures underlying high-dimensional neural activity.
arXiv:2606. 20431v1 Announce Type: new Abstract: Continual learning (CL) systems often forget previously acquired knowledge, yet the mechanisms driving forgetting remain hard to isolate in practice because real datasets entangle many factors.
arXiv:2606. 00732v1 Announce Type: new Abstract: Learning long-range non-stationary temporal patterns remains a core challenge for modern sequence models, particularly in strict streaming settings.
arXiv:2610.19997v1 Announce Type: new Abstract: Efficient sequence modeling remains a fundamental challenge in machine learning. Despite the success of Transformers, recurrent neural networks (RNNs)...
arXiv:2609.38356v1 Announce Type: new Abstract: Dynamical Systems Reconstruction (DSR) aims to infer models from observed time series that reproduce a system's qualitative long-term behavior. Continu...
arXiv:2602. 01196v2 Announce Type: replace Abstract: Recurrent neural policies are widely used in partially observable control and meta-RL tasks.