arXiv:2606. 02365v1 Announce Type: cross Abstract: Shampoo is attracting considerable attention for its superior performance on large-scale optimization benchmarks; yet it faces a significant practical bottleneck: the prohibitive computational overhead of matrix inversion.
arXiv:2602. 02016v2 Announce Type: replace Abstract: Shampoo is one of the leading approximate second-order optimizers: a variant of it has won the MLCommons AlgoPerf competition, and it has been shown to produce models with lower activation outliers that are easier to compress.
The article surveys recent neural‑network optimizers, noting that the field has moved beyond simple Adam variants to encompass matrix‑ and layer‑level designs, time‑policy horizons, and state representations that survive sharding and low‑precision computation. It categorizes optimizers along four axes—temporal estimation, update geometry, horizon management, and representation & systems—highlighting methods such as Muon, Shampoo, SOAP, and quantized states. The survey concludes that while matrix‑aware methods are a genuine advance, no single optimizer universally replaces AdamW, and performance depends on model scale, data‑to‑parameter ratio, batch size, schedule, partitioning, tuning budget, and target metric.
arXiv:2608.29448v1 Announce Type: cross Abstract: Physics-informed neural networks (PINNs) often face ill-conditioned objectives that limit high-accuracy training. Dense quasi-Newton methods improve...
arXiv:2610.08534v1 Announce Type: new Abstract: Understanding the principles behind Shampoo has recently guided the development of more effective neural network optimizers. These methods learn a prec...
arXiv:2603.28678v2 Announce Type: replace Abstract: We introduce PACE, a backpropagation-free continual test-time adaptation system that directly optimizes the affine parameters of normalization laye...
arXiv:2606. 31390v1 Announce Type: cross Abstract: Low-rank matrix optimization is often carried out via the Burer-Monteiro (BM) formulation, but choosing the factorization rank $r$ is delicate and can substantially slow optimization.
arXiv:2511. 19716v3 Announce Type: replace-cross Abstract: Stochastic Gradient Descent (SGD) often slows in the late stage of training due to anisotropic curvature and gradient noise.
arXiv:2509. 08765v4 Announce Type: replace-cross Abstract: Data-driven acceleration of scientific computing workflows has been a high-profile aim of machine learning (ML) for science, with numerical simulation of transient partial differential equations (PDEs) being one of the main applications.
arXiv:2609.36692v1 Announce Type: cross Abstract: Matrix optimizers have emerged as a promising direction, with Muon standing out as a prominent design. Revisiting Muon through its full-Gram represen...
The paper introduces JANUS, a post‑hoc weight rectification framework that enforces Parameter Space Orthogonality to prevent catastrophic forgetting when fine‑tuning foundation models. By projecting updates into the Jacobian Null Space and employing a Multi‑step Adaptive Rectification mechanism, JANUS dynamically verifies trust regions and adjusts step sizes. Additional techniques such as ghost projection, ghost orientation comparison, and sequence‑level SVD compression provide temporal and spatial efficiency, enabling JANUS to integrate seamlessly with various fine‑tuning methods and effectively mitigate the stability‑plasticity dilemma.
arXiv:2607. 26247v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized.
arXiv:2606. 12337v1 Announce Type: cross Abstract: Inverse problems governed by partial differential equations (PDEs) are central to computational mechanics and are commonly solved by adjoint-based optimization, while physics-informed neural networks (PINNs) have emerged as a flexible alternative.