Hugging Face Trending Papers
Jul 7
Efficient Long-Horizon Learning for Learned Optimization
Learned optimization aims to improve upon hand-designed optimizers (e. g.
Efficient Long-Horizon Learning for Learned Optimization
arXiv:2607. 06772v1 Announce Type: new Abstract: Learned optimization aims to improve upon hand-designed optimizers (e.
Related stories
arXiv AI
Jul 22
Soft-TransFormers for Continual Learning
arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.
arXiv AI
3d ago
Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...
arXiv Machine Learning
Jun 11
On the Stability of Growth in Structural Plasticity
arXiv:2605. 15435v2 Announce Type: replace Abstract: Standard deep-learning pipelines usually choose the network architecture before training and keep it fixed throughout optimization.
arXiv AI
Jun 15
Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining
arXiv:2601. 22108v2 Announce Type: replace-cross Abstract: Continued pretraining is optimized with fixed self-supervised tasks but selected by downstream performance, creating a coarse feedback loop in which practitioners evaluate checkpoints, change data mixtures or objectives, and restart runs, while individual updates remain blind to target capabilities.
arXiv AI
Sep 18
Green-ELM: Efficient Analytic Learning via High-Dimensional Random Projections
arXiv:2604. 15613v4 Announce Type: replace-cross Abstract: We present Green-ELM, a non-iterative neural architecture that replaces gradient-based optimization of the output layer with a closed-form analytic solution over a fixed, high-dimensional random feature representation.
arXiv AI
Jul 7
OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
arXiv:2607. 04033v1 Announce Type: cross Abstract: Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented.
arXiv AI
Jun 15
Gefen: Optimized Stochastic Optimizer
arXiv:2606. 13894v1 Announce Type: cross Abstract: AdamW is a default optimizer for modern deep learning, but its first and second moment states add roughly two parameter-sized buffers to training memory.
arXiv Machine Learning
Jul 20
Learning Faster without Deeper Networks: A*-Inspired Batch Selection for Efficient CNN Training
arXiv:2607. 15745v1 Announce Type: new Abstract: Common practice when training Convolutional Neural Networks (CNNs) is to use randomly shuffled mini-batches.
arXiv AI
Jun 18
From Memorization to Parameter Interference: How Overtraining Experts Harms Model Merging
arXiv:2506. 14126v2 Announce Type: replace-cross Abstract: Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets.
arXiv Machine Learning
Jun 5
Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss
arXiv:2606. 06418v1 Announce Type: new Abstract: Many modern applications of deep learning involve training a neural network via a one-step prediction loss (e.
arXiv Machine Learning
Jul 10
Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning
arXiv:2603. 12478v2 Announce Type: replace-cross Abstract: Multimodal instruction tuning is often compute-inefficient because training budgets are spread across large mixed image-video pools whose utility is highly uneven.