arXiv Machine Learning
Jul 28
Greedy dynamical meta-learning
arXiv:2607. 23925v1 Announce Type: new Abstract: Gradient descent scales well to large models, but becomes unstable over long time horizons.
Reptile: A scalable meta-learning algorithm
We’ve developed a simple meta-learning algorithm called Reptile which works by repeatedly sampling a task, performing stochastic gradient descent on it, and updating the initial parameters towards the final parameters learned on that task. Reptile is the application of the Shortest Descent algorithm to the meta-learning setting, and is mathematically similar to first-order MAML (which is a version of the well-known MAML algorithm) that only needs black-box access to an optimizer such as SGD or Adam, with similar computational efficiency and performance.
Related stories
Hugging Face Trending Papers
Jul 27
Greedy dynamical meta-learning
Gradient descent scales well to large models, but becomes unstable over long time horizons. Gradient-free optimizers can scale to arbitrary timespans, but are hobbled by high dimensions.
arXiv Machine Learning
6d ago
A New First-Order Meta-Learning Algorithm with Convergence Guarantees
arXiv:2409. 03682v2 Announce Type: replace Abstract: Learning new tasks by leveraging prior experience is a fundamental trait of intelligent systems.
OpenAI Blog
Mar 8, 2018
On first-order meta-learning algorithms
arXiv Machine Learning
Jul 9
Efficient Long-Horizon Learning for Learned Optimization
arXiv:2607. 06772v1 Announce Type: new Abstract: Learned optimization aims to improve upon hand-designed optimizers (e.
Hugging Face Trending Papers
Jul 7
Efficient Long-Horizon Learning for Learned Optimization
Learned optimization aims to improve upon hand-designed optimizers (e. g.
arXiv Machine Learning
Aug 10
Optimization as a Dynamical System: Generative Schedules from Latent ODEs
arXiv:2509. 23052v2 Announce Type: replace Abstract: We present a new meta-learning method to determine the optimal learning rate schedule for gradient descent.
arXiv AI
Jul 22
From Trajectories to Instructions: Language-Conditioned Meta-Reinforcement Learning
arXiv:2607. 18830v1 Announce Type: cross Abstract: Model-Agnostic Meta-Learning (MAML) is a widely used framework for reinforcement learning (RL) that enables efficient transfer by learning global policy parameters that can be rapidly adapted to new tasks.
arXiv Machine Learning
Jun 17
MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization
arXiv:2606. 17526v1 Announce Type: new Abstract: Efficient optimization is essential for training large language models.
arXiv Machine Learning
1d ago
Metaplasticity as adaptive gradient preconditioning for incremental learning
arXiv:2608. 14634v1 Announce Type: new Abstract: Biological intelligence naturally prevents catastrophic forgetting through Complementary Learning Systems (CLS) theory, a macroscopic consolidation process driven at the local level by synaptic metaplasticity: the continuous, history-dependent neuromodulation of individual synapses.
arXiv Machine Learning
Jul 7
Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses
arXiv:2406. 14340v2 Announce Type: replace-cross Abstract: The standard stochastic gradient descent (SGD) optimization method, as well as adaptive methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (particularly, in the situation of constant learning rates).
arXiv Machine Learning
Aug 11
Beyond the Capability Boundary: Zeroth-Order Optimization for Self-Evolving LLM Agents
arXiv:2608. 09292v1 Announce Type: new Abstract: Self-evolving methods improve the capabilities of LLM agents by sampling trajectories from the underlying LLMs and learning from these trajectories.