arXiv:2609.36240v1 Announce Type: new Abstract: Does representation learning stop when the training loss stops improving? We study this question for matrix Muon, whose polar-normalised updates have a...
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
arXiv:2610.01728v1 Announce Type: cross Abstract: Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple...
arXiv:2608. 06766v1 Announce Type: cross Abstract: Training changes a network's predictions while allocating task-relevant structure across its internal units.
The paper investigates how the geometry of teacher neural networks affects the learnability of student networks in teacher‑student setups. By formalizing learnability as the success rate of reaching the global minimum, the authors identify two teacher distributions—one maximizing node dissimilarity (easy) and one minimizing it (hard)—that lead to markedly different success rates across various settings and activation functions. They analyze the loss landscape of small networks, revealing two types of suboptimal local minima (out‑of‑bounds and interior) whose attraction regions depend on teacher structure, and demonstrate that adjusting learning rates for the readout layer and inner biases can improve success rates. whyItMatters:"The study highlights that teacher geometry, often overlooked, plays a crucial role in determining how effectively a student network can learn, offering guidance for designing more realistic teacher‑student experiments."
arXiv:2607. 23346v1 Announce Type: new Abstract: Modern deep neural networks are potent catalysts for scientific and industrial impact, yet excessive parameter counts impede deployment in low-compute settings such as hospital equipment and energy infrastructure.
The paper introduces the "lift" technique for training input‑convex neural networks, replacing the traditional non‑negative weight constraint enforced by projected gradient descent or a softplus map. By adding a learnable slack variable and an unconstrained network that processes a permutation‑invariant batch summary, the lift couples batch‑dependent latent weights to the gradient, increasing update variance and enabling faster escape from the softplus shoulder. Experiments show that when the softplus method stalls at the shoulder, the lift achieves tighter fits and reconstructs targets roughly three times faster, while both methods agree when the shoulder is rarely reached.
arXiv:2512. 24780v2 Announce Type: replace Abstract: Neural networks trained with standard objectives exhibit behaviors characteristic of probabilistic inference: soft clustering, prototype specialization, and Bayesian uncertainty tracking.
arXiv:2607. 08843v1 Announce Type: new Abstract: In artificial and biological neural networks, concepts are often encoded as consistent linear directions in representation space.
arXiv:2609. 12994v1 Announce Type: new Abstract: Heavy-tailed empirical spectral densities of neural-network weight matrices are widely used as diagnostics of implicit self-regularization, but the step complexity of heavy-tail emergence remains poorly understood.
arXiv:2609.08381v1 Announce Type: cross Abstract: Equivariant networks are commonly trained with Adam, yet recent work reports that matrix-structured optimizers such as Muon can perform better on the...
arXiv:2609.40127v1 Announce Type: cross Abstract: Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keepin...