arXiv AI October 19, 2026 By Wu S. Zihan, Brian A. Yuan, Guillaume Bellec

Local proxies for memory-efficient training

The paper proposes a memory‑efficient training method for transformers by training a local proxy network for each block to predict the global output via distillation. This allows gradients to be propagated locally while closely approximating the true backpropagation gradients, achieving a better accuracy‑memory trade‑off on CIFAR‑100 vision transformers. The approach also improves fine‑tuning of large language models under tight memory budgets, outperforming other memory‑efficient zeroth‑order algorithms.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 15

DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation

arXiv:2506. 14202v4 Announce Type: replace-cross Abstract: End-to-end backpropagation requires storing activations throughout all layers, creating memory bottlenecks that limit model scalability.

By Makoto Shing, Masanori Koyama, Takuya Akiba

llms diffusion

arXiv Machine Learning

Jun 9

Rethinking Local Learning: A Cheaper and Faster Recipe for LLM Post-Training

arXiv:2605. 04913v4 Announce Type: replace-cross Abstract: LLM post-training typically propagates task gradients through the full depth of the model.

By Hengyu Shi, Tianyang Han, Peizhe Wang, Zhiling Wang, Xu Yang, Junhao Su

llms

Hugging Face Trending Papers

Aug 11

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting.

llms efficiency benchmarks safety

Hugging Face Trending Papers

Aug 9

Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods

AI training's rising resource intensity is straining electricity supplies and carbon budgets, motivating systematic study of memory-efficient training on constrained hardware. We benchmark five gradient optimizers (SGD, Adam, Adagrad, Adadelta, and Conjugate Gradient Descent) under three memory strategies (standard training, gradient checkpointing, and gradient accumulation) across four transformer architectures (ViT, ModernBERT, Llama 3.

llms computer-vision multimodal benchmarks

arXiv AI

Oct 27

Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design

arXiv:2610.04867v2 Announce Type: replace-cross Abstract: Greedy block-wise local learning splits a network into gradient-isolated blocks trained by local auxiliary losses, deleting the backward pass...

By Jihwan Moon, Sheir A. Zaheer, Jinmyoung Lee, Gunhee Kim, Chan Y. Park

llms computer-vision

arXiv Machine Learning

Jun 5

SpanNorm: Reconciling Training Stability and Performance in Deep Transformers

arXiv:2601. 22580v2 Announce Type: replace-cross Abstract: The success of Large Language Models (LLMs) hinges on the stable training of deep Transformer architectures.

By Chao Wang, Bei Li, Jiaqi Zhang, Xinyu Liu, Yuchun Fan, Linkun Lyu, Xin Chen, Jingang Wang, Tong Xiao, Peng Pei, Xunliang Cai