arXiv Machine Learning
Sparse Competition during Training For the Emergence of Specialized Modules
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
5d ago
FlowGenesis: Representation-Driven Self-Organization of Neural Architectures
FlowGenesis is a framework that lets neural networks self‑organize their architecture during training by adding or removing neurons based on a local statistic called Representation Flux. Neurons with high Flux are split to preserve function, while low‑Flux, low‑activity neurons are pruned. Experiments on image and text classification show that networks starting from different sizes converge to similar neuron counts and performance, and the method also improves continual learning by reducing parameters for new tasks without sacrificing accuracy.
arXiv AI
Oct 12
Neural networks are more modular than single neurons suggest
The paper argues that neural networks can exhibit modular organization even when individual neurons are not specialized. It introduces a nested hierarchy of modularity—unit-level, orthogonal, and linear—and shows that many vision, language, and RNN models lack unit-level modularity but still display clear functional modularity at the subspace level. The authors demonstrate that these subspaces can be causally validated through ablation and align with brain data, revealing distinct content/style or syntax/semantics separations.
arXiv AI
Jun 30
Distilling a Modular Reservoir Through a Genomic Bottleneck
arXiv:2606. 28380v1 Announce Type: cross Abstract: The intricate structures of biological neural networks largely emerge during development, guided by a comparatively compressed blueprint encoded in the genome.
arXiv Machine Learning
Jun 30
A Mechanistic Study of Transformers Training Dynamics
arXiv:2410. 24050v3 Announce Type: replace Abstract: Large-scale pretraining of transformers has been central to the success of foundation models.
arXiv AI
Oct 26
Sparsity-driven emergence of functional modularity in multitask neural networks
The paper demonstrates that applying sparsity-inducing regularization to multitask neural networks encourages the emergence of functional modularity without predefining module boundaries. By profiling multilayer perceptrons and a lightweight transformer, the authors identify compact, task‑selective subnetworks—called functional tokens—that can be selectively retained to preserve specific task performance while degrading others. These tokens tend to form in deeper layers, and their compactness and specialization are influenced by regularization strength, form, and task weighting, while they can be recombined to reconstruct composite computations without retraining.
arXiv Machine Learning
Sep 30
Traversing the solution space of neural networks with Hessian Null Space Continuation
arXiv:2609.38081v1 Announce Type: new Abstract: On a single task, deep networks can learn many solutions, depending on their optimizer, training data, architecture, and hyperparameters. Many of these...