← Back to all news
arXiv Machine Learning

Leveraging Dale's Principle as an Inductive Bias in Recurrent Neural Networks

The paper introduces EISep RNN, a recurrent neural network that enforces Dale's principle—each neuron is either exclusively excitatory or inhibitory—through a learnable excitatory‑inhibitory separation. Experiments across multi‑task learning, continual learning, and reinforcement learning show that EISep yields more stable training, competitive or improved performance, and sparser, more modular connectivity compared to vanilla RNNs and prior Dale‑constrained models. The architecture also produces stable and informative latent dynamics, demonstrating that Dale's principle can serve as a biologically grounded inductive bias for RNNs.

Related stories

arXiv AI
Sep 25

ELiSe: Efficient Learning of Sequences in Structured Recurrent Networks

The paper introduces ELiSe, a model that leverages cortical network scaffolds and dendritic compartments to learn complex non‑Markovian spatio‑temporal patterns using only local, always‑on, phase‑free synaptic plasticity. It demonstrates the model’s ability to acquire and replay intricate sequences, exemplified by a birdsong learning mock‑up, and shows robustness to external disturbances and flexibility in parameter settings.

By Laura Kriener, Kristin V\"olk, Ben von H\"unerbein, Federico Benitez, Walter Senn, Mihai A. Petrovici
More like this →
arXiv AI
Jun 18

Structured Representation Learning with Locally Linear Embeddings and Adaptive Feature Fusion

arXiv:2606. 18469v1 Announce Type: cross Abstract: Neuroscientific research has revealed that the brain encodes complex behaviors by leveraging structured, low-dimensional manifolds and dynamically fusing multiple sources of information through adaptive gating mechanisms.

By Somjit Nath, Jackson J Cone, Derek Nowrouzezahrai, Samira Ebrahimi Kahou
ragreinforcement-learningbenchmarks
More like this →
arXiv AI
Oct 7

Continuous Memory Machines

The paper introduces the Continuous Memory Machine (CMM), a recurrent neural network that uses separate matrix-valued short‑term and long‑term memory states. Short‑term memory tracks recent neural activity with neuron‑level models for rapid computation, while long‑term memory stores information for later use; both are jointly updated by a Transformer that allows bidirectional read‑write operations. Experiments on algorithmic, in‑context learning, and recurrent reasoning tasks show that CMM outperforms many baselines and generalizes better than previous memory‑augmented networks, while maintaining interpretable attention patterns from the Continuous Thought Machine.

By Ciaran Regan, Kai Arulkumaran, Luke Darlow, Stefania Druga, Sebastian Risi, Llion Jones
llms
More like this →
arXiv AI
Jun 24

Topological Neural Dynamics: A Neuron-wise Framework for Sequence Modeling

arXiv:2606. 21295v2 Announce Type: replace-cross Abstract: Existing sequence models, including RNNs, LSTMs, continuous-time networks, and Transformers, share a common structural principle: layer-wise dynamics, where all neurons in the same layer co-evolve through a shared parameterized operator, leaving individual neurons no freedom to evolve independently.

By Borui Cai, Yao Zhao
llmssafety
More like this →