The paper introduces a finite‑width geometric framework that explains how learned feature geometries are organized, transported, and selectively aligned in deep neural networks. It quantifies incompatibilities among weight‑generated covariance, gates, and backward sensitivities using three families of commutators, and provides exact layerwise identities that decompose these commutators into sources such as downstream transport, adjacent‑layer imbalance, and nonlinear gate‑covariance interactions. The study demonstrates that spectral alignment is a layer‑ and scale‑dependent compatibility phenomenon governed by transport, interaction, cancellation, and possible damping, rather than a universal consequence of training.
By Kaj Nystr\"om
arXiv:2610.32849v1 Announce Type: cross
Abstract: During joint training, backpropagation allows later layers to reshape earlier representations, a feedback mechanism absent from purely layer-wise lea...
By Yatin Dandi, Matteo Vilucchio, Hugo Tabanelli, Florent Krzakala
The paper investigates how exact automatic differentiation behaves under hard‑ReLU gradient descent. It shows that while gradient‑descent states converge to the piecewise‑smooth gradient flow, the derivative of the training map does not, due to missing event‑time sensitivities captured by saltation matrices. The study demonstrates that for convex objectives, activation events can create large sensitivity gaps, and provides empirical evidence that event‑aware corrections are necessary for accurate flow derivatives.
By Xiaoyang Li, Runni Zhou
In implicit neural representations (INRs), 3D signals are encoded directly in network weights, making stable weight-space operations essential. However, permutation symmetry causes independently train...
The paper introduces a progressive permutation alignment technique for implicit neural representations, using layer-wise correlation estimation, Hungarian matching, and temperature-controlled spectral annealing to gradually sharpen the alignment toward a permutation matrix. This deterministic, lightweight method avoids costly gradient optimization and is followed by a final refinement stage that optimizes layer-by-layer weight matching. Experiments on MLPs, CNNs, and NeRFs demonstrate reliable neuron correspondence recovery and reduced loss barriers during weight interpolation.
By Yuta Horioka, Naoya Chiba, Yuki Uranishi
The paper analyzes gradient flow in a two‑layer linear neural network as the input, hidden, and output dimensions grow proportionally. Using a closed‑form solution combined with random matrix theory, it derives exact asymptotic spectral distributions for the weight matrices at all times and tracks how the learned subspace evolves. In the limit of small initialization, the study reveals a silent alignment phase where singular directions align quickly while singular values and loss stay unchanged, followed by a visible learning phase on a logarithmic timescale where singular values increase and loss decreases through a saddle‑to‑saddle dynamic that becomes continuous in the high‑dimensional limit.
By Simon Martin (DI-ENS, LPENS, SIERRA), Francis Bach (DI-ENS, SIERRA), Giulio Biroli (LPENS)
The paper introduces a unified matrix‑spectral framework that treats deep neural networks as data‑dependent products of linear operators, enabling the analysis of stability and interpretability through spectral quantities. It defines a Global Matrix Stability Index that aggregates spectral information from Jacobians, parameter gradients, Neural Tangent Kernel operators, and loss Hessians into a single stability scale, and shows that spectral entropy provides a more realistic measure of sensitivity than worst‑case bounds. Experiments on MNIST, CIFAR‑10, and CIFAR‑100 demonstrate that modest spectral regularization can significantly improve attribution stability without large changes to global spectral summaries.
By Ronald Katende
arXiv:2603. 13751v2 Announce Type: replace Abstract: Physics-informed neural networks (PINNs) have achieved notable success in modeling dynamical systems governed by partial differential equations (PDEs).
By Zhangyong Liang, Huanhuan Gao
arXiv:2607. 17696v1 Announce Type: cross Abstract: We develop an adjoint-sensitivity framework for positional influence in causal residual Transformers and separate unconditional analytic results from conditional boundary-shape conclusions.
By Cheng Huan, Hongwei Yuan
The paper introduces COFM, a framework for consistent optimal transport flow matching that uses partially input convex neural networks (PICNN) to parameterize the transport potential. By adding a Hamilton‑Jacobi residual to the training objective, COFM enforces dynamical consistency and supports both one‑step transport and multi‑step ODE sampling without costly inner optimization. Experiments on benchmark datasets show that COFM achieves competitive performance while reducing L^2‑UVP by over 2× and cutting computational time by about 9× compared to state‑of‑the‑art models.
By Fanghui Song, Zhongjian Wang, Jiebao Sun
arXiv:2606. 06171v1 Announce Type: cross Abstract: Physics-Informed Neural Networks inherently suffer from task interference because they rely on a shared parameter space to satisfy both governing differential equations and boundary conditions.
By Cornelius Otchere, Michael Shields
The paper investigates how gradient descent behaves near codimension‑one bifurcations in recurrent neural networks by analyzing the global empirical Neural Tangent Kernel (GeNTK). Under local center‑manifold conditions, the parameter‑to‑state Jacobian is approximated by a low‑rank normal‑form operator, causing the GeNTK and Fisher information matrix to become strongly amplified and anisotropic, concentrating on a rank‑one or rank‑two channel depending on the bifurcation type. Experiments on high‑dimensional RNNs confirm that this low‑rank concentration coincides with abrupt loss changes, subtask interference, and aligns with changes in memory dynamics in a 15‑task LeakyRNN.
By James Hazelden, Eric Shea-Brown