The paper revisits magnitude pruning for recurrent neural networks, arguing that it neglects the flow of information between neurons. It proposes modeling RNN dependencies as a partially ordered set (poset) and pruning by retaining the meet‑irreducible elements of the resulting dependency lattice, a method that requires a well‑defined dependency order obtained by condensing thresholded graphs into strongly connected components. Experiments on 128‑unit LSTMs trained on sequential MNIST show that the earlier critical‑set criterion fails to preserve any recurrent weights, that the new order‑based score performs no better than simple metrics like degree or PageRank, and that enforcing acyclicity from initialization incurs significant accuracy loss at high sparsity levels. whyItMatters":"The study demonstrates that without imposing a proper dependency structure, pruning recurrent networks can inadvertently remove all recurrent connections, highlighting the need for careful theoretical foundations in sparsity‑aware training."
arXiv:2608. 05464v1 Announce Type: cross Abstract: The pruning of network connections is key to brain function but, despite its importance, there exist few biologically-plausible pruning rules with demonstrated good performance.
The pruning of network connections is key to brain function but, despite its importance, there exist few biologically-plausible pruning rules with demonstrated good performance. In this work we evaluate noise-prune, a recently introduced unsupervised local pruning rule for recurrent networks that uses noisy fluctuations to determine the importance of connections.
The paper introduces Dynamic Activity-Dependent Pruning (DADP), a biologically inspired pruning technique that uses accumulated pre‑synaptic activations and post‑synaptic error gradients to assess connection importance. Unlike traditional methods that rely on fixed layer budgets or magnitude thresholds, DADP employs a single global threshold, allowing it to dynamically allocate sparsity across a network’s depth and prune neurons and channels during training. Experiments on MLP, VGG‑16, ResNet‑18, BiLSTM‑CRF, and MiniBERT show that DADP matches or surpasses existing pruning methods, achieving 73.67% accuracy at 99% sparsity on ResNet‑18 while preserving feature diversity as measured by Shannon entropy and effective rank.
The paper investigates how single‑hidden‑layer MLPs can fit training data yet fail to recover the underlying rule, focusing on higher‑order interactions and nuisance inputs. Using synthetic parity tasks, the authors benchmark different optimizers (SGD, Adam, Muon) and show that while all achieve perfect accuracy on second‑order interactions, performance drops sharply for higher orders, with Muon outperforming the others at fourth order. Experiments also reveal that freezing or removing nuisance‑related weights dramatically alters training outcomes, highlighting the role of nuisance learning in shaping the rules a shallow network can represent.
arXiv:2603. 02234v3 Announce Type: replace-cross Abstract: The Strong Lottery Ticket Hypothesis (SLTH) states that large, randomly initialized neural networks contain sparse subnetworks capable of approximating a target function at initialization without training, suggesting that pruning alone is sufficient.
LILA (Latent-Informed Layer Analysis) introduces a calibration‑free method for structured pruning of large language models by scoring neuron importance using the Kolmogorov–Smirnov distance between singular value distributions of full and neuron‑ablated feed‑forward network weight matrices. The approach requires no training, calibration data, or auxiliary networks, and outperforms existing methods such as PruneNet and SliceGPT on LLaMA‑2‑7B and Phi‑2 at various sparsity levels. After a single epoch of LoRA fine‑tuning, LILA matches heavily calibrated baselines, and a Neural Tangent Kernel analysis provides theoretical support for its spectral importance criterion. Additionally, LILA can dynamically allocate sparsity budgets, achieving state‑of‑the‑art generative preservation and revealing architectural bottlenecks at higher compression.
arXiv:2311.04902v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurt...
arXiv:2606. 12278v1 Announce Type: cross Abstract: Neural network pruning reduces model size by removing less important parameters while aiming to preserve predictive performance.
arXiv:2609.24401v1 Announce Type: new Abstract: Structured pruning is a model compression technique that is used to reduce the computational cost of deploying deep neural networks on resource-constra...
Neural network pruning reduces model size by removing less important parameters while aiming to preserve predictive performance. Although the Lottery Ticket Hypothesis (LTH) shows that sparse subnetworks can match dense networks when trained from suitable initializations, its iterative pruning procedure requires multiple complete training cycles.
arXiv:2610. 01369v1 Announce Type: new Abstract: Understanding a nonlinear dynamical system from time series requires not only reproducing its trajectories, but also identifying a simple representation that preserves its essential dynamical structure.
arXiv:2602. 24266v2 Announce Type: replace-cross Abstract: Which internal mechanisms of a neural network can be replaced while preserving the computation it performs?