arXiv AI
Jun 3
q0: Primitives for Hyper-Epoch Pretraining
arXiv:2606. 03938v1 Announce Type: cross Abstract: Multi-epoch training is becoming the standard now that compute is growing faster than the supply of high-quality text.
Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection
The paper introduces Repeated Optimizer Resampling (ROR), a method that treats optimizer choice as a hyperparameter and searches for the best optimizer during a single training run. ROR periodically scouts each candidate optimizer for a short number of epochs, then continues training with the best scout, allowing the optimizer to change over time. Experiments on MNIST, Fashion‑MNIST, and motor insurance claim‑count models show that one‑epoch ROR uses only 24–35% of the training effort required to exhaustively evaluate all optimizers while achieving comparable performance.
Related stories
arXiv Machine Learning
Jul 9
Efficient Long-Horizon Learning for Learned Optimization
arXiv:2607. 06772v1 Announce Type: new Abstract: Learned optimization aims to improve upon hand-designed optimizers (e.
arXiv Machine Learning
Jun 11
Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning
arXiv:2606. 12278v1 Announce Type: cross Abstract: Neural network pruning reduces model size by removing less important parameters while aiming to preserve predictive performance.
arXiv Machine Learning
Jul 20
Learning Faster without Deeper Networks: A*-Inspired Batch Selection for Efficient CNN Training
arXiv:2607. 15745v1 Announce Type: new Abstract: Common practice when training Convolutional Neural Networks (CNNs) is to use randomly shuffled mini-batches.
Hugging Face Trending Papers
Jun 10
Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning
Neural network pruning reduces model size by removing less important parameters while aiming to preserve predictive performance. Although the Lottery Ticket Hypothesis (LTH) shows that sparse subnetworks can match dense networks when trained from suitable initializations, its iterative pruning procedure requires multiple complete training cycles.
arXiv Machine Learning
Aug 4
AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization
arXiv:2608. 01997v1 Announce Type: new Abstract: Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on.
Hugging Face Trending Papers
Jul 7
Efficient Long-Horizon Learning for Learned Optimization
Learned optimization aims to improve upon hand-designed optimizers (e. g.
arXiv Machine Learning
Jun 2
Iterated Population Based Training with Task-Agnostic Restarts
arXiv:2511. 09190v2 Announce Type: replace Abstract: Hyperparameter Optimization (HPO) can lift the burden of tuning hyperparameters (HPs) of neural networks.
arXiv Machine Learning
Jul 31
Towards Stability of Parameter-Free Optimization
arXiv:2405. 04376v4 Announce Type: replace Abstract: Hyperparameter tuning, particularly the selection of an appropriate learning rate in adaptive gradient training methods, remains a challenge.
arXiv Machine Learning
Jul 27
Hyperball May Not Be a Free Lunch
arXiv:2607. 22444v1 Announce Type: new Abstract: For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates.
arXiv AI
Jun 9
Structured Neuron Pruning in Deep Neural Networks Using Multi-Armed Bandits
arXiv:2606. 07615v1 Announce Type: cross Abstract: Deep neural networks often contain redundant hidden units.
arXiv AI
Jul 14
Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection
arXiv:2603. 28921v3 Announce Type: replace-cross Abstract: The critical damping condition of the damped harmonic oscillator model of SGD with momentum (Qian, 1999) yields a momentum schedule with no tuned hyperparameters: mu(t) = 1 - 2*sqrt(alpha(t)).