arXiv:2606. 29554v1 Announce Type: new Abstract: Shuffle order can be a larger source of fine-tuning noise than a memoryless analysis predicts: fixed-clock optimizer memory makes local equal-multiset contrasts first order in the learning rate rather than second order, and the resulting order channel can be large enough for a single seed to flip a close A/B comparison.
arXiv:2608.23816v1 Announce Type: new Abstract: Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We pres...
arXiv:2607. 02893v1 Announce Type: new Abstract: Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width.
arXiv:2608.23601v1 Announce Type: cross Abstract: EDA flow parameter tuning is critical for quality-of-results~(QoR), yet the parameter space is large, tightly coupled, and full evaluations are prohi...
arXiv:2608. 01041v1 Announce Type: cross Abstract: Machine-learning predictors estimate processor performance far faster than cycle-level simulation.
arXiv:2610.16451v1 Announce Type: cross Abstract: Fine-tuning large language models at scale is increasingly limited by the memory cost of backpropagation. Zeroth-Order (ZO) optimizers reduce this bu...
arXiv:2607. 16268v1 Announce Type: new Abstract: Adaptive optimizers such as Adam and AdamW apply the same update rule regardless of whether training is in a chaotic early phase or near convergence.
arXiv:2608. 19491v1 Announce Type: new Abstract: Most modern optimizers form their momentum as an exponential moving average (EMA) of past gradients, forgetting every direction at one fixed rate.
arXiv:2607. 14516v1 Announce Type: new Abstract: Interpreting optimizers as gradient-flow discretizations has motivated applying higher-order Runge-Kutta (RK) integrators to neural networks.
arXiv:2608. 04407v1 Announce Type: cross Abstract: Memory-efficient matrix optimizers such as Sinkhorn gradient descent remove most AdamW optimizer state for dense Transformer matrices, but direct application to Mixture-of-Experts (MoE) training is unreliable.
The paper introduces Activation-Keyed Momentum (AK‑Momentum), a momentum update that uses the input activation of a linear layer as a key to apply a delta‑rule update, allowing each direction to decay at a rate proportional to its frequency of appearance. AK‑Momentum is proven to be a valid momentum, incorporates input‑side curvature correction without matrix inversion, and clears stale directions faster than traditional exponential moving average (EMA) under both fixed and drifting optima. It can replace the momentum buffer of any optimizer, scales with width under μP, adds only 22–25% extra compute, and demonstrates significant step‑count reductions in FineWeb‑Edu pretraining and other benchmarks. whyItMatters":"AK‑Momentum offers a principled, efficient way to adapt momentum decay to anisotropic training dynamics, improving convergence speed and stability across a range of models and datasets."
The paper presents a method for reliably scheduling operator execution in real‑time inference on small devices, achieving exact peak‑memory predictions and certified optimality. By profiling the generated program and modeling its memory usage, the authors can permute statements to match the modeled peak, guaranteeing bitwise‑identical results on CPUs and characterizing GPU limitations. They also provide efficient lower‑bound traps and analyze selective recomputation, showing it is NP‑hard in general but can reduce peak memory on certain architectures.
The paper studies how different optimizers perform as training duration (overtraining) increases, focusing on matrix‑preconditioned methods (Muon, SOAP) and a momentum‑scheduled method (ADANA) compared to AdamW. Across models ranging from 51M to 253M parameters and overtraining factors up to 256×, the authors find that optimal learning‑rate schedules, weight‑decay coefficients, and memory settings shift with horizon, and that ADANA consistently outperforms AdamW, especially with log‑time weight decay and momentum cooldown. Muon and SOAP maintain roughly constant token‑efficiency advantages, with SOAP potentially improving at the highest overtraining levels.