arXiv Machine Learning
CAST: History-Adaptive Spectral Clipping for LLM Pretraining
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 24
Robust and Fast Training via Per-Sample Clipping
arXiv:2605. 02701v2 Announce Type: replace-cross Abstract: We propose a robust gradient estimator based on per-sample gradient clipping and analyze its properties both theoretically and empirically.
arXiv Machine Learning
Jun 10
AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping
arXiv:2502. 11034v3 Announce Type: replace Abstract: Loss spikes remain a persistent obstacle in large-scale language model pretraining.
arXiv Machine Learning
Oct 23
Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?
arXiv:2605. 27733v2 Announce Type: replace Abstract: Training instabilities such as loss spikes are frequently the result of stochastic gradient noise.
arXiv Machine Learning
Aug 4
GradientStabilizer:Fix the Norm, Not the Gradient
arXiv:2502. 17055v5 Announce Type: replace Abstract: Training instability in modern deep learning systems is frequently triggered by rare but extreme gradient-norm spikes, which can induce oversized parameter updates, corrupt optimizer state, and lead to slow recovery or divergence.
arXiv Machine Learning
Aug 14
Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation?
arXiv:2608. 12332v1 Announce Type: cross Abstract: In recent years, low-rank adaptation (LoRA) has emerged as a significant paradigm that freezes pre-trained weights and introduces small, learnable adapters instead of fine-tuning the full set of parameters.
arXiv Machine Learning
Oct 9
Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights
arXiv:2610. 11730v1 Announce Type: new Abstract: Standard weight decay treats each weight matrix as a vector and ignores its spectral structure.