← Back to all news
arXiv AI

Sparsity-driven emergence of functional modularity in multitask neural networks

The paper demonstrates that applying sparsity-inducing regularization to multitask neural networks encourages the emergence of functional modularity without predefining module boundaries. By profiling multilayer perceptrons and a lightweight transformer, the authors identify compact, task‑selective subnetworks—called functional tokens—that can be selectively retained to preserve specific task performance while degrading others. These tokens tend to form in deeper layers, and their compactness and specialization are influenced by regularization strength, form, and task weighting, while they can be recombined to reconstruct composite computations without retraining.

Related stories

arXiv Machine Learning
Sep 22

The Ups and Downs of Backprop Weights

The paper discusses how backpropagation enables deep learning but does not inherently organize parameters for reusable functional components, leading to weight entanglement where overlapping parameter sets hinder independent modification. It introduces weight operators—parameterized modules that can be composed at inference—to address this, proposing a two-stage learning process that first infers operator composition and then updates only the selected operators. Vector Networks (VNs) are presented as an implementation that couples operator selection to local error-driven updates, demonstrating that learned operators can be recombined in unseen ways while keeping updates confined to the relevant parameter sets.

By Giuseppe Chindemi, Benjamin F. Grewe
More like this →
arXiv AI
Oct 12

Neural networks are more modular than single neurons suggest

The paper argues that neural networks can exhibit modular organization even when individual neurons are not specialized. It introduces a nested hierarchy of modularity—unit-level, orthogonal, and linear—and shows that many vision, language, and RNN models lack unit-level modularity but still display clear functional modularity at the subspace level. The authors demonstrate that these subspaces can be causally validated through ablation and align with brain data, revealing distinct content/style or syntax/semantics separations.

By Shreya Saha, Benjamin Bergen, Meenakshi Khosla
llmsbenchmarkssafety
More like this →
arXiv Machine Learning
Aug 31

More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations

The paper introduces Mixture of Activations (MoA), a token‑adaptive feedforward network design that mixes multiple activation functions using lightweight gates while sharing linear projections. It also presents learnable activations (LA) as an input‑independent variant. The authors theoretically prove that MoA strictly surpasses both fixed‑activation FFNs and LA in expressive power, and empirically demonstrate that MoA achieves lower loss and better scaling on dense and MoE language models from 0.12 B to 2 B parameters with minimal overhead.

By Mingze Wang, Jinbo Wang, Yikuan Xia, Kai Shen, Shu Zhong
llms
More like this →
arXiv AI
Jun 17

Dimensionality Controls When Modularity Helps in Continual Learning

arXiv:2606. 17889v1 Announce Type: cross Abstract: Compositional learning systems must balance plasticity, the ability to acquire new knowledge, with stability, the preservation of previously learned components, especially when tasks share structure and risk interference.

By Kathrin Korte, Christian Medeiros Adriano, Joachim Winther Pedersen, Eleni Nisioti, Sebastian Risi
safety
More like this →
arXiv AI
Oct 1

Effective Does Not Mean Useful: Conditional Functional Substitutability for Redundancy and Scaling in Transformers

The paper introduces Conditional Functional Substitutability (CFS) as a new way to measure redundancy in Transformers by examining when intermediate states produce similar downstream responses. CFS uncovers functional relationships and potential reductions that traditional importance- or similarity-based metrics miss, revealing systematic reorganization as models scale. Experiments across modalities and Transformer families show that performance gains do not always align with increased substitutability, and that models with more independent functional structure perform better, offering a functional explanation for diminishing returns and enabling more efficient dynamic computation.

By Jiaheng Chen, Jiaxing Li, Yucheng Xiao, Xinyong Cai, Juncheng Bu, Lan Yu, Tinghe Zhang
llms
More like this →