arXiv AI
Parallelism Strategy Chaining for Fast Training Convergence
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 9
Efficient Scaling of LLM Training with Flexible Context Parallelism
arXiv:2602. 21788v2 Announce Type: replace-cross Abstract: Scaling long-context capabilities is crucial for Large Language Models (LLMs).
arXiv Machine Learning
Aug 4
Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training
arXiv:2608. 02391v1 Announce Type: cross Abstract: Tool-using large language model (LLM) agents produce long, multi-turn trajectories, making gradient-based post-training memory-intensive.
arXiv Machine Learning
Aug 27
Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
The paper investigates why the orthogonal optimiser Muon outperforms Adam in large language model pretraining by analysing the spectral properties of Transformer loss landscapes. It finds that Muon’s momentum buffers exhibit an anisotropic spectral profile with a volatile head and a tolerant bulk, enabling larger effective step sizes. Building on this insight, the authors propose Spectral‑Aware Muon (SAMuon) and a lightweight variant, which adjust the bulk scaling while keeping the head unchanged, achieving 13–24 % fewer training tokens than Muon without extra FLOPs.
arXiv Machine Learning
4d ago
Probe-Space Preconditioning for Fast and Stable Zero-Order Training
arXiv:2609.38095v1 Announce Type: new Abstract: Backpropagation (BP) dominates deep learning but imposes a massive memory tax. For example, training OPT-30B with Adam requires $\approx$ 600GB of GPU...
arXiv Machine Learning
Jul 14
Memory Savings at What Cost? A Study of Alternatives to Backpropagation
arXiv:2506. 21833v2 Announce Type: replace Abstract: Forward-mode automatic differentiation (FmAD) and zero-order (ZO) optimization are increasingly proposed as memory-efficient, backpropagation-free alternatives for large language model (LLM) fine-tuning, yet their benefits are typically evaluated only against standard backpropagation (BP), omitting memory-efficient variants such as activation checkpointing.
arXiv Machine Learning
Jun 2
CRePE: Convolution-aware Relative Importance in Post-training Pruning with Efficient Search
arXiv:2606. 01544v1 Announce Type: new Abstract: Deploying Large Language Models (LLMs) in practice incurs substantial memory and computational costs.