arXiv:2607. 11666v1 Announce Type: new Abstract: Grokking is a phenomenon in which neural networks initially memorize training data and only later exhibit strong generalization after prolonged optimization.
arXiv:2606. 00073v1 Announce Type: cross Abstract: We investigate how internal representations emerge across hierarchical processing systems by introducing a neuroscience-inspired framework for analyzing deep spiking neural networks (SNN) through the lens of functional connectivity.
The paper demonstrates that applying sparsity-inducing regularization to multitask neural networks encourages the emergence of functional modularity without predefining module boundaries. By profiling multilayer perceptrons and a lightweight transformer, the authors identify compact, task‑selective subnetworks—called functional tokens—that can be selectively retained to preserve specific task performance while degrading others. These tokens tend to form in deeper layers, and their compactness and specialization are influenced by regularization strength, form, and task weighting, while they can be recombined to reconstruct composite computations without retraining.
The paper introduces Dynamic Activity-Dependent Pruning (DADP), a biologically inspired pruning technique that uses accumulated pre‑synaptic activations and post‑synaptic error gradients to assess connection importance. Unlike traditional methods that rely on fixed layer budgets or magnitude thresholds, DADP employs a single global threshold, allowing it to dynamically allocate sparsity across a network’s depth and prune neurons and channels during training. Experiments on MLP, VGG‑16, ResNet‑18, BiLSTM‑CRF, and MiniBERT show that DADP matches or surpasses existing pruning methods, achieving 73.67% accuracy at 99% sparsity on ResNet‑18 while preserving feature diversity as measured by Shannon entropy and effective rank.
The paper proposes a biologically inspired micro‑sleep technique called napping for recurrent spiking neural networks, combining proportional weight scaling with continuous stochastic membrane activity. Experiments on an unsupervised SNN trained with trace‑based STDP on Gabor‑preprocessed MNIST show that well‑tuned napping can match the classification accuracy of conventional weight normalization while offering different clustering characteristics. The study suggests that napping may be preferable when representational structure is more important than raw classification speed, despite its higher simulation cost.
The paper introduces fundamental representational units (FRUs) in neural networks and proposes two evaluation criteria—fidelity and stability—to assess them. It demonstrates that threshold-activated sparse autoencoders (TSAEs) can identify FRUs with high fidelity and stability, supported by large-scale experiments. The study further shows that more stable FRUs are more monosemantic and can be combined across layers to reconstruct semantic structures that align with human conceptual organization.
arXiv:2606. 14975v1 Announce Type: cross Abstract: How the wiring and functional organization of cortex shape recurrent computation remains a central question in both neuroscience and machine learning.
arXiv:2602.23405v3 Announce Type: replace-cross Abstract: Varying the number of neurons while preserving function is difficult because neural networks' dense connectivity creates strong computational...
The paper demonstrates that using a single activation function across all nodes in artificial neural networks imposes hard limits on evolutionary search, particularly for sparse evolved substrates. By evolving per-node activation functions from an 18-function palette, the authors show that oscillatory functions can solve parity problems at all tested scales, while monotonic functions fail beyond the simplest case. The study reveals that the choice of activation functions, beyond topology and weights, critically influences what evolutionary search can achieve, and that heterogeneous assignments discovered via indirect encoding are unlikely to be selected manually.
arXiv:2606. 30444v1 Announce Type: cross Abstract: Neural networks are known to be susceptible to over-reliance on spurious correlations.
The paper introduces the Dormant Spiking Neuron (DSN), a state-driven mechanism that allows neurons in spiking neural networks to enter a dormant state when their membrane potential falls below a negative threshold. This dynamic gating prunes redundant computations at the instance level and is trained end‑to‑end via backpropagation through time with surrogate gradients. Experiments on static and neuromorphic datasets, including ImageNet, show that DSN reduces computational energy by 40% to over 70% and improves baseline SNN accuracy by 1.34%, outperforming state‑of‑the‑art static pruning methods.
arXiv:2606. 03935v1 Announce Type: cross Abstract: The ability to train spiking neural networks is essential for modeling biological neural networks as well as for neuromorphic computing.