arXiv Machine Learning
Jun 17
MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization
arXiv:2606. 17526v1 Announce Type: new Abstract: Efficient optimization is essential for training large language models.
GradPower: Powering Gradients for Faster Language Model Pre-Training
arXiv:2505. 24275v4 Announce Type: replace Abstract: We propose GradPower, a lightweight gradient-transformation technique for accelerating language model pre-training.
Related stories
arXiv AI
Jun 15
Gefen: Optimized Stochastic Optimizer
arXiv:2606. 13894v1 Announce Type: cross Abstract: AdamW is a default optimizer for modern deep learning, but its first and second moment states add roughly two parameter-sized buffers to training memory.
arXiv Machine Learning
Aug 11
Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods
arXiv:2608. 08961v1 Announce Type: new Abstract: AI training's rising resource intensity is straining electricity supplies and carbon budgets, motivating systematic study of memory-efficient training on constrained hardware.
Hugging Face Trending Papers
Aug 9
Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods
AI training's rising resource intensity is straining electricity supplies and carbon budgets, motivating systematic study of memory-efficient training on constrained hardware. We benchmark five gradient optimizers (SGD, Adam, Adagrad, Adadelta, and Conjugate Gradient Descent) under three memory strategies (standard training, gradient checkpointing, and gradient accumulation) across four transformer architectures (ViT, ModernBERT, Llama 3.
arXiv Machine Learning
Jul 9
Efficient Long-Horizon Learning for Learned Optimization
arXiv:2607. 06772v1 Announce Type: new Abstract: Learned optimization aims to improve upon hand-designed optimizers (e.
arXiv Machine Learning
Jun 9
Convergence Bound and Critical Batch Size of Muon Optimizer
arXiv:2507. 01598v5 Announce Type: replace Abstract: Muon, a recently proposed optimizer that leverages the inherent matrix structure of neural network parameters, has demonstrated strong empirical performance, indicating its potential as a successor to standard optimizers such as AdamW.
Hugging Face Trending Papers
Jul 7
Efficient Long-Horizon Learning for Learned Optimization
Learned optimization aims to improve upon hand-designed optimizers (e. g.
arXiv AI
Jul 23
In-Run Data Shapley for Adam Optimizer
arXiv:2602. 00329v4 Announce Type: replace-cross Abstract: Reliable data attribution is essential for mitigating bias and reducing computational waste in modern machine learning, with the Shapley value serving as the theoretical gold standard.
arXiv Machine Learning
Jun 11
Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning
arXiv:2601. 04710v2 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) achieves strong performance but is often limited by the memory overhead of backpropagation.
arXiv AI
Jul 14
Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization
arXiv:2607. 09967v1 Announce Type: cross Abstract: Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps.
arXiv Machine Learning
1d ago
On the Principles Behind Neural Network Optimizers
arXiv:2608. 16760v1 Announce Type: new Abstract: Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation.
arXiv AI
Aug 7
Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging
arXiv:2608. 05541v1 Announce Type: new Abstract: Evolution Strategy (ES) is a promising alternative to gradient-based fine-tuning for resource-constrained Large Language Model (LLM) reasoning.