The paper introduces the Drift Contract, a spectral update geometry for local learning that improves depth robustness and hyperparameter stability. By applying momentum orthogonalization with spectral step scaling to per‑layer updates, the authors achieve consistent performance across a wide range of widths and depths on CIFAR‑10 MLPs, outperforming local Adam and providing a per‑layer, input‑conditioned drift bound. The study also shows that the spectral geometry itself, rather than step‑size rules, drives the observed depth robustness, while a negative result indicates that the stability benefit is limited to non‑normalized layers.
By Fabien Polly
arXiv:2606. 25971v1 Announce Type: new Abstract: Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object.
By Alexander H\"agele, Alejandro Hern\'andez-Cano, Atli Kosson, Martin Jaggi
arXiv:2605. 01928v2 Announce Type: replace Abstract: We optimize losses that jump: spiking thresholds, quantized layers, and discrete routing put jumps in the forward pass, where backpropagation does not apply.
By An T. Le
The paper introduces the "lift" technique for training input‑convex neural networks, replacing the traditional non‑negative weight constraint enforced by projected gradient descent or a softplus map. By adding a learnable slack variable and an unconstrained network that processes a permutation‑invariant batch summary, the lift couples batch‑dependent latent weights to the gradient, increasing update variance and enabling faster escape from the softplus shoulder. Experiments show that when the softplus method stalls at the shoulder, the lift achieves tighter fits and reconstructs targets roughly three times faster, while both methods agree when the shoulder is rarely reached.
By Ali Siahkoohi
arXiv:2608.08322v2 Announce Type: replace-cross
Abstract: Adaptive loss-balancing schemes for physics-informed neural networks rest on a premise that every residual should be driven to zero. For leve...
By Muhammad Akbar Khan
arXiv:2607. 18343v1 Announce Type: cross Abstract: Federated fine-tuning is bottlenecked by communication: FedAvg and pseudo-gradient schemes transmit a payload that scales with the model, and gradient compression shrinks it by only a constant factor.
By Radhakrishna Achanta, Will Reed