The paper introduces Sparse Sum-of-Squares (SOS) Layers, a new type of differentiable layer that reuses the sparse structure of forward semidefinite programming (SDP) solves during backpropagation. By avoiding dense certificate construction, the method achieves up to 2.7× faster warm‑call times and reduces memory usage from 160 GiB to 1.7 GiB on large problems. Experiments demonstrate its effectiveness across sparse polynomial certification, control, combinatorial relaxations, robotics, oscillator chains, and pose‑graph synchronization, achieving zero certified violations on unseen instances with far fewer optimization solves than finite‑difference approaches.
Second-order cone programs (SOCPs) are a fundamental class of convex optimization problems widely applied across engineering domains. Classical interior-point methods are robust and accurate, but thei...
arXiv:2609.24021v1 Announce Type: cross Abstract: Neural operators are data-driven models that learn mappings from inputs that parameterize partial differential equations, such as spatially varying c...
arXiv:2607. 22467v1 Announce Type: new Abstract: Data scarcity poses a fundamental challenge in training generative models to produce initial guesses for parametric optimization problems that are otherwise numerically expensive to solve.
The paper introduces Shallow Universal Polynomial Networks (SUPNs), a single‑hidden‑layer architecture that replaces standard activations with learnable multivariate polynomials, thereby increasing expressivity per unit. SUPNs achieve universal approximation by either widening the network or increasing polynomial complexity, and the authors prove convergence rates and quasi‑optimal parameter choices. Empirical tests across a wide range of smooth, nonsmooth, and discontinuous functions in up to ten dimensions show that SUPNs often outperform deep neural networks and Kolmogorov‑Arnold networks in accuracy and variability while requiring fewer parameters and lower training cost.
Neural operators are data-driven models that learn mappings from inputs that parameterize partial differential equations, such as spatially varying coefficients, initial conditions, forcing terms, bou...
arXiv:2608. 12655v1 Announce Type: new Abstract: A flat training curve does not reveal whether a neural network has reached a global optimum, is locally trapped, is representation-limited, or is mismatched to its trainer.
arXiv:2605. 26854v2 Announce Type: replace Abstract: The scalable solution of large sparse linear systems is a bottleneck in scientific computing and graph analysis.
arXiv:2609.10311v1 Announce Type: cross Abstract: The lottery ticket hypothesis posits the existence of winning tickets: sparse subnetworks that, when trained in isolation from their original initial...
arXiv:2106. 06998v5 Announce Type: replace Abstract: Training convolutional neural networks at scale demands substantial memory, largely because intermediate activations must be stored for backpropagation.
arXiv:2609.38095v1 Announce Type: new Abstract: Backpropagation (BP) dominates deep learning but imposes a massive memory tax. For example, training OPT-30B with Adam requires $\approx$ 600GB of GPU...
arXiv:2608. 06912v1 Announce Type: new Abstract: The top-$k$ operation is a fundamental building block of modern sparse computation, enabling token routing, expert activation, memory selection, and attention pruning.
BLOKS is a Conjugate Gradient-based solver for Nyström‑sketched Kernel Ridge Regression that introduces a novel preconditioner combining block Jacobi approximation with sketching. This preconditioner’s memory cost is independent of dataset size and model parameters, enabling scaling to large models while maintaining GPU acceleration. Experiments on standard large‑scale datasets show BLOKS achieves accuracy comparable to or better than state‑of‑the‑art kernel solvers with favorable time/memory complexity.