arXiv AI
Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 19
Local proxies for memory-efficient training
The paper proposes a memory‑efficient training method for transformers by training a local proxy network for each block to predict the global output via distillation. This allows gradients to be propagated locally while closely approximating the true backpropagation gradients, achieving a better accuracy‑memory trade‑off on CIFAR‑100 vision transformers. The approach also improves fine‑tuning of large language models under tight memory budgets, outperforming other memory‑efficient zeroth‑order algorithms.
arXiv Machine Learning
Jun 9
Rethinking Local Learning: A Cheaper and Faster Recipe for LLM Post-Training
arXiv:2605. 04913v4 Announce Type: replace-cross Abstract: LLM post-training typically propagates task gradients through the full depth of the model.
arXiv Machine Learning
1d ago
Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima
The paper introduces Nexus, an optimizer that encourages gradient similarity during pretraining to bring task‑specific minima closer together. Experiments on models from 130M to 3B parameters show that Nexus achieves the same pretraining loss as standard optimizers while significantly improving downstream performance, including a 15% accuracy boost on GSM8k and a 0.012 reduction in out‑of‑distribution loss for the 3B model.
arXiv AI
Oct 15
Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models
arXiv:2610.17590v1 Announce Type: cross Abstract: Diffusion language models (DLMs) have attracted wide attention for their fast, parallel decoding, but they are believed to be far less efficient to t...
arXiv Machine Learning
Jul 14
Memory Savings at What Cost? A Study of Alternatives to Backpropagation
arXiv:2506. 21833v2 Announce Type: replace Abstract: Forward-mode automatic differentiation (FmAD) and zero-order (ZO) optimization are increasingly proposed as memory-efficient, backpropagation-free alternatives for large language model (LLM) fine-tuning, yet their benefits are typically evaluated only against standard backpropagation (BP), omitting memory-efficient variants such as activation checkpointing.
arXiv Machine Learning
Jun 5
Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss
arXiv:2606. 06418v1 Announce Type: new Abstract: Many modern applications of deep learning involve training a neural network via a one-step prediction loss (e.