arXiv Machine Learning
Biologically Inspired Mechanisms for Facilitating Grokking in Multilayer Perceptrons
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jul 14
How to Tame Grokking: Representation Geometry as a Control Signal
arXiv:2607. 11666v1 Announce Type: new Abstract: Grokking is a phenomenon in which neural networks initially memorize training data and only later exhibit strong generalization after prolonged optimization.
arXiv AI
Jun 2
Rare Events, Real Signals: Functional Ensembles as Units of Computation in Deep Spiking Networks
arXiv:2606. 00073v1 Announce Type: cross Abstract: We investigate how internal representations emerge across hierarchical processing systems by introducing a neuroscience-inspired framework for analyzing deep spiking neural networks (SNN) through the lens of functional connectivity.
arXiv AI
Oct 26
Sparsity-driven emergence of functional modularity in multitask neural networks
The paper demonstrates that applying sparsity-inducing regularization to multitask neural networks encourages the emergence of functional modularity without predefining module boundaries. By profiling multilayer perceptrons and a lightweight transformer, the authors identify compact, task‑selective subnetworks—called functional tokens—that can be selectively retained to preserve specific task performance while degrading others. These tokens tend to form in deeper layers, and their compactness and specialization are influenced by regularization strength, form, and task weighting, while they can be recombined to reconstruct composite computations without retraining.
arXiv Machine Learning
Oct 9
DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method
The paper introduces Dynamic Activity-Dependent Pruning (DADP), a biologically inspired pruning technique that uses accumulated pre‑synaptic activations and post‑synaptic error gradients to assess connection importance. Unlike traditional methods that rely on fixed layer budgets or magnitude thresholds, DADP employs a single global threshold, allowing it to dynamically allocate sparsity across a network’s depth and prune neurons and channels during training. Experiments on MLP, VGG‑16, ResNet‑18, BiLSTM‑CRF, and MiniBERT show that DADP matches or surpasses existing pruning methods, achieving 73.67% accuracy at 99% sparsity on ResNet‑18 while preserving feature diversity as measured by Shannon entropy and effective rank.
arXiv Machine Learning
Sep 15
Exploring napping paradigm for Recurrent Spiking Neural Networks
The paper proposes a biologically inspired micro‑sleep technique called napping for recurrent spiking neural networks, combining proportional weight scaling with continuous stochastic membrane activity. Experiments on an unsupervised SNN trained with trace‑based STDP on Gabor‑preprocessed MNIST show that well‑tuned napping can match the classification accuracy of conventional weight normalization while offering different clustering characteristics. The study suggests that napping may be preferable when representational structure is more important than raw classification speed, despite its higher simulation cost.
arXiv AI
1d ago
Representational Atoms in Neural Networks
The paper introduces fundamental representational units (FRUs) in neural networks and proposes two evaluation criteria—fidelity and stability—to assess them. It demonstrates that threshold-activated sparse autoencoders (TSAEs) can identify FRUs with high fidelity and stability, supported by large-scale experiments. The study further shows that more stable FRUs are more monosemantic and can be combined across layers to reconstruct semantic structures that align with human conceptual organization.