The paper introduces a new approach to solving the MAX‑CUT problem by combining Particle Polynomial Relaxation (PPR) with a spectral graph‑coarsening technique called Combinatorial Multigrid (CMG). CMG provides fast, structure‑aware initial solutions that warm‑start PPR, enabling the algorithm to escape many local optima inherent in the binary optimization landscape. Experiments on benchmark instances show that the CMG+PPR pipeline achieves over 97 % of the known optimum on average, reaches the exact optimum on all sixty Biq Mac instances when augmented with discrete local‑search, and scales to graphs with up to 20,000 vertices while dramatically reducing memory usage.
arXiv:2609.38095v1 Announce Type: new Abstract: Backpropagation (BP) dominates deep learning but imposes a massive memory tax. For example, training OPT-30B with Adam requires $\approx$ 600GB of GPU...
The paper introduces TACO, a new optimizer for fine‑tuning large language models that drastically reduces optimizer state memory while preserving first‑order gradients. TACO selects the sign of the largest magnitude entry in each column of weight matrices, achieving a 174× reduction in persistent optimizer memory compared to AdamW8bit and a 2.9× decrease in peak training memory on OPT‑13B. This allows full‑parameter fine‑tuning of 30–32B‑parameter models on a single 80 GB GPU across multiple model families and tasks, with comparable accuracy and runtime to existing methods.
arXiv:2606. 10445v1 Announce Type: new Abstract: Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup.
Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning.
The paper introduces a hardware-software co‑design framework that compresses Mixture‑of‑Experts (MoE) model weights into low‑precision, hardware‑native sparse representations, enabling efficient execution on Sparse Tensor Cores (SpTCs). By relaxing discrete support selection through continuous reparameterization, the method jointly optimizes quantized weights and supports a router‑weighted reconstruction objective, achieving up to 4.35 percentage‑point gains in joint sparse‑quantization accuracy while retaining 96.09% of the original model’s performance. A custom grouped sparse GEMM kernel further boosts inference speed, outperforming NVIDIA’s baseline by up to 1.65× and reducing latency by up to 4.03× on B200 GPUs.
arXiv:2608. 06912v1 Announce Type: new Abstract: The top-$k$ operation is a fundamental building block of modern sparse computation, enabling token routing, expert activation, memory selection, and attention pruning.
arXiv:2606. 13894v1 Announce Type: cross Abstract: AdamW is a default optimizer for modern deep learning, but its first and second moment states add roughly two parameter-sized buffers to training memory.
arXiv:2605. 26854v2 Announce Type: replace Abstract: The scalable solution of large sparse linear systems is a bottleneck in scientific computing and graph analysis.
arXiv:2610.04204v2 Announce Type: replace Abstract: Training large language models (LLMs) entails a fundamental trade-off: memory-efficient optimizers such as Adam discard cross-parameter curvature,...
Clean is a memory‑efficient optimizer that uses Nyström sketching to approximate SOAP’s curvature preconditioners, reducing memory complexity from quadratic to linear while preserving rich curvature information. It also reintegrates off‑subspace components and offers a low‑precision variant, Q‑Clean, that cuts optimizer memory by over 50% compared to Muon while maintaining strong performance. Clean outperforms AdamW in wall‑clock time and enables pre‑training a 13B‑parameter model on a single 80GB GPU.
Decoupled Prune-and-Reconstruct (DPR) introduces a double‑layer sparsification strategy for graph neural networks that first prunes nodes to reduce memory usage and then reconstructs the full graph during inference. By training on a compact core of 30% of the nodes, DPR achieves 96.9% of the accuracy of a dense model while using only 1,450 MiB of peak memory on a single 6 GB GPU, outperforming edge‑sparsification baselines and reducing training memory by 50.1%. The method scales to millions of nodes, enabling all‑node inference on VRAM‑limited hardware without incurring a permanent accuracy loss.
arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.