The Topological Trouble With Transformers
arXiv:2604. 17121v3 Announce Type: replace Abstract: Transformers encode structure in sequences via an expanding contextual history.
arXiv:2604. 17121v3 Announce Type: replace Abstract: Transformers encode structure in sequences via an expanding contextual history.
The paper introduces the Continuous Memory Machine (CMM), a recurrent neural network that uses separate matrix-valued short‑term and long‑term memory states. Short‑term memory tracks recent neural activity with neuron‑level models for rapid computation, while long‑term memory stores information for later use; both are jointly updated by a Transformer that allows bidirectional read‑write operations. Experiments on algorithmic, in‑context learning, and recurrent reasoning tasks show that CMM outperforms many baselines and generalizes better than previous memory‑augmented networks, while maintaining interpretable attention patterns from the Continuous Thought Machine.
arXiv:2605. 13438v3 Announce Type: replace Abstract: Existing agent memory remains predominantly reactive and retrieval-based, lacking the capacity to autonomously organize experience into persistent cognitive structure.
The paper proposes a second‑order recurrent neural network that replaces traditional neuron‑to‑neuron communication with a spatially evolving field governed by discretized partial differential equations. This design creates a structured infinite‑order RNN where each state depends on its entire history, providing an unbounded receptive field while keeping the number of parameters fixed. The authors derive conditions for marginal stability that prevent vanishing or exploding gradients, and demonstrate that the architecture outperforms other recurrent models on long‑horizon benchmarks with fewer parameters.
arXiv:2609.06006v1 Announce Type: cross Abstract: Deep learning for time series has progressed through successive architectural paradigms, from recurrent networks and transformers to structured state...
arXiv:2607. 11958v1 Announce Type: new Abstract: Under the free energy principle, a predictive system does not observe reality directly; it maintains a generative model of the world and experiences that model's best current hypothesis.
arXiv:2608. 14634v1 Announce Type: new Abstract: Biological intelligence naturally prevents catastrophic forgetting through Complementary Learning Systems (CLS) theory, a macroscopic consolidation process driven at the local level by synaptic metaplasticity: the continuous, history-dependent neuromodulation of individual synapses.
The paper introduces ELiSe, a model that leverages cortical network scaffolds and dendritic compartments to learn complex non‑Markovian spatio‑temporal patterns using only local, always‑on, phase‑free synaptic plasticity. It demonstrates the model’s ability to acquire and replay intricate sequences, exemplified by a birdsong learning mock‑up, and shows robustness to external disturbances and flexibility in parameter settings.
arXiv:2602. 01196v2 Announce Type: replace Abstract: Recurrent neural policies are widely used in partially observable control and meta-RL tasks.
arXiv:2609.16053v1 Announce Type: cross Abstract: Long-term memory is essential for LLM-based agents operating over extended interactions. Existing memory systems primarily update memory when new inf...
arXiv:2608.30946v1 Announce Type: new Abstract: Closed-loop human-AI systems generate high-dimensional behavioural trajectories whose collective dynamics remain obscure. Using 297,915 learners' adapt...
MePo++ is a post‑training framework designed for general continual learning (GCL) that unifies representation refinement and reconciliation. It introduces MetaPrep, which enhances representation plasticity via unsupervised meta‑refinement on pseudo continual sequences, and StreamAlign, which maintains stability by reconciling online features with a stable pretrained geometry. Experiments across various pretrained models, datasets, and continual learning baselines show that MePo++ consistently improves performance in PTM‑based GCL.