The paper investigates a negative weight drift that arises from the interaction between standard loss functions (MSE or cross‑entropy) and positively biased activation functions. This drift, which is inherent to the optimization process rather than the data, pushes downstream weights toward negative values during early training and leads to high activation sparsity—up to nearly 90% in GPT‑nano when using ReLU. The authors also examine the impact of this sparsity, noting that model performance degrades sharply when activation sparsity exceeds about 70%, and study sparsity‑inducing activation functions that produce spikes in intermediate transformer layers.
arXiv:2610.14500v1 Announce Type: cross Abstract: Many widely used activation functions, including ReLU, GELU, Swish, and Sigmoid, produce predominantly positive outputs, resulting in a nonzero post-...
Deep neural networks can exhibit periodic loss spikes known as the "Slingshot Mechanism" during long‑term training. This paper shows that such spikes arise from finite‑precision floating‑point arithmetic: when the correct‑class logit becomes much larger than the others, the gradient for that class is rounded to zero while others remain nonzero, breaking the zero‑sum constraint and creating a positive feedback loop that inflates feature and classifier norms. The authors term this Numerical Feature Inflation (NFI) and demonstrate that it explains the rapid norm growth, reappearance of gradients, and loss spikes, while also showing that partial absorption can still cause hidden norm growth without visible spikes.
arXiv:2610.02598v1 Announce Type: new Abstract: Deploying LLMs on consumer-grade GPUs with insufficient memory to hold their weights can result in prohibitively slow inference, because decoding repea...
arXiv:2607. 21005v1 Announce Type: new Abstract: Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable.
The paper demonstrates that neural networks can maintain perfect training accuracy while their training cross‑entropy remains nearly flat at a positive level, a phenomenon observed on modular addition tasks. Using ReLU networks with supported‑Polar hidden updates, normalized output‑layer updates, and weight decay, the authors show that hidden features evolve and held‑out accuracy can rise from zero to above 90% even as cross‑entropy stays flat. Theoretical results for one‑hidden‑layer networks and empirical evidence from deeper networks and transformers confirm that correct classification need not coincide with minimal cross‑entropy, challenging conventional assumptions about loss plateaus.
The paper investigates intermittent spikes in loss observed during the pre‑training of large language models. It demonstrates that simple quadratic models with heavy‑tailed, power‑law distributed inputs can reproduce these spikes, and introduces a theoretical framework and simulation algorithm to model per‑layer training dynamics of deep neural networks. In a case study using an Adam configuration with β2 close to 1, the simulation predicts spiky behavior from a checkpoint and suggests a β2 schedule that empirically improves performance.
arXiv:2607. 23777v1 Announce Type: cross Abstract: The discovery of scaling laws has motivated training neural networks on ever increasing quantities of data.
The paper introduces Dynamic Activity-Dependent Pruning (DADP), a biologically inspired pruning technique that uses accumulated pre‑synaptic activations and post‑synaptic error gradients to assess connection importance. Unlike traditional methods that rely on fixed layer budgets or magnitude thresholds, DADP employs a single global threshold, allowing it to dynamically allocate sparsity across a network’s depth and prune neurons and channels during training. Experiments on MLP, VGG‑16, ResNet‑18, BiLSTM‑CRF, and MiniBERT show that DADP matches or surpasses existing pruning methods, achieving 73.67% accuracy at 99% sparsity on ResNet‑18 while preserving feature diversity as measured by Shannon entropy and effective rank.
arXiv:2607. 29503v1 Announce Type: new Abstract: While neural networks are typically evaluated by their training and test performance, these metrics do not reveal how robust a learned representation is.
arXiv:2606. 26538v1 Announce Type: cross Abstract: Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value.
arXiv:2502. 11034v3 Announce Type: replace Abstract: Loss spikes remain a persistent obstacle in large-scale language model pretraining.
arXiv:2606. 25971v1 Announce Type: new Abstract: Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object.