arXiv:2504. 06407v2 Announce Type: replace Abstract: Machine Unlearning aims to remove undesired information from trained models without full retraining from scratch.
arXiv:2607. 23970v1 Announce Type: cross Abstract: Machine Unlearning aims to remove undesired information from trained models without full retraining from scratch.
arXiv:2609.06862v1 Announce Type: new Abstract: Superposition refers to neural networks representing more features than they have dimensions. It offers a possible explanation for polysemantic neurons...
Machine Unlearning aims to remove undesired information from trained models without full retraining from scratch. Despite recent progress, the loss landscape and optimization geometry of unlearning are poorly understood.
arXiv:2609.24379v1 Announce Type: cross Abstract: Mechanistic interpretability of vision transformers seeks to decompose model computation into human-readable units, but learned representations entan...
arXiv:2608.30366v1 Announce Type: new Abstract: The loss landscape of Deep Neural Networks (DNNs) exhibits highly complex and non-convex properties. Recent studies have revealed the phenomenon of mod...
arXiv:2604. 00230v2 Announce Type: replace Abstract: Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised.
Mode Dynamics is a framework that links coordinated parameter motion during neural network training to changes in model outputs. By decomposing recorded parameter trajectories into collective modes and mapping each mode to its state‑dependent output effect, the framework uses decomposition, reconstruction, and concentration mechanisms to analyze how behavior emerges from parameter updates. Experiments on CIFAR‑100 autoencoding and a Pythia‑70M pretraining trajectory show that a small set of leading modes can reconstruct behavior, that induced effects depend on the training path, and that modal complexity varies with task and data.
arXiv:2608. 06597v1 Announce Type: cross Abstract: A scientific theory of deep learning, comprising learning dynamics and statistical properties of learned models, is rapidly gaining attention.
arXiv:2610.31783v1 Announce Type: new Abstract: We study the early training dynamics of nonlinear residual networks with blocks of zero hidden layers as a model of more general residual networks. We...
The paper studies how Adam’s two momentum timescales, β1 and β3, influence loss spikes during neural‑network training. By mapping training dynamics across the (β1,β3) plane, the authors find an approximately linear boundary, 1-β3 = C(1-β1), that separates spiky from non‑spiky behavior, with the coefficient C linked to the effective loss exponent in superquadratic loss functions. They also show that confident cross‑entropy losses create a core–wall landscape that behaves superquadratically at the scale of an optimizer update, explaining the observed spikes.
Mechanistic interpretability of vision transformers seeks to decompose model computation into human-readable units, but learned representations entangle many concepts in each neuron. Feature superposi...