The paper introduces TACO, a new optimizer for fine‑tuning large language models that drastically reduces optimizer state memory while preserving first‑order gradients. TACO selects the sign of the largest magnitude entry in each column of weight matrices, achieving a 174× reduction in persistent optimizer memory compared to AdamW8bit and a 2.9× decrease in peak training memory on OPT‑13B. This allows full‑parameter fine‑tuning of 30–32B‑parameter models on a single 80 GB GPU across multiple model families and tasks, with comparable accuracy and runtime to existing methods.

By Jichao Jiang (University of Central Florida), Cristian McGee (University of Central Florida), El Houcine Bergou (Mohammed VI Polytechnic University), Hanqin Cai (University of Central Florida), Aritra Dutta (University of Central Florida)

llms fine-tuning

arXiv Machine Learning

Aug 4

AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization

arXiv:2608. 01997v1 Announce Type: new Abstract: Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on.

By Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana, Gopi Krishna Nedanuri

benchmarks

One email a morning, machine-written

Related stories

Make LLM Fine-tuning 2x faster with Unsloth and 🤗 TRL

Accelerating PyTorch distributed fine-tuning with Intel technologies

HighTide: An Agent-Curated Open-Source VLSI Benchmark Suite

Parameter-Efficient Fine-Tuning using 🤗 PEFT

TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization