arXiv Machine Learning
Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jul 10
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
arXiv:2607. 07953v1 Announce Type: cross Abstract: Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context.
arXiv AI
Jul 29
Stable FP4 Training via Transposition-Invariant Block Quantization
arXiv:2607. 24953v1 Announce Type: cross Abstract: Reducing training precision is a key lever for improving the e ciency of large language model (LLM) training, but pushing beyond FP8 to 4-bit oating point (FP4) remains challenging due to instability during optimization.
arXiv Machine Learning
Sep 22
FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention
FlashBoB introduces an I/O‑efficient algorithm for exact backward‑over‑backward (BoB) in softmax attention, enabling precise second‑order differentiation without large intermediate tensors. By exploiting a hierarchical affine structure, the method confines computation to on‑chip tiles and limits off‑chip memory traffic, achieving θ(N² d²/M) HBM usage. Experiments show FlashBoB scales to sequence lengths of 262K on a single A100 GPU, outperforming prior exact baselines and FlashBack by up to 6.3×.
arXiv Computation and Language
Sep 1
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
arXiv:2608.30320v1 Announce Type: new Abstract: We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and a...
arXiv AI
Jul 21
LaCache: Exact Caching and Precision-Adaptive Inference for Diffusion Large Language Models
arXiv:2607. 16339v1 Announce Type: new Abstract: Diffusion-based Large Language Models(DLLMs) enable parallel generation via Semi-Autoregressive (SAR) decoding in text generation.
arXiv AI
Aug 18
Forward Pass Domain Adaptation (Without Cross-Layer Backpropagation)
arXiv:2608. 14563v1 Announce Type: cross Abstract: Forward-Pass-Only MLP training (FPO) adapts large language models without a backward pass through the model body, achieving 2.