arXiv Machine Learning
Affinity-Aware Sharding for Delayed Tensor Parallelism
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jun 9
Scaling Neural Network Verification with Tensor Parallelism and Fully Sharded Data Parallelism
arXiv:2606. 09377v1 Announce Type: cross Abstract: Formal neural network verification -- proving that a network satisfies safety properties for \emph{all} inputs in a specified domain -- is bounded in practice by GPU memory: standard implementations of bound-propagation algorithms (IBP, CROWN, $\alpha$-CROWN) require weight and relaxation-coefficient matrices to reside entirely on one accelerator.
Hugging Face Trending Papers
Jun 8
Scaling Neural Network Verification with Tensor Parallelism and Fully Sharded Data Parallelism
Formal neural network verification -- proving that a network satisfies safety properties for \emph{all} inputs in a specified domain -- is bounded in practice by GPU memory: standard implementations of bound-propagation algorithms (IBP, CROWN, $α$-CROWN) require weight and relaxation-coefficient matrices to reside entirely on one accelerator. We adapt two parallelism techniques originally developed for large-scale model training to the \texttt{auto\_LiRPA}\,/\,$α,β$-CROWN verification framework.
arXiv Machine Learning
1d ago
Scaling Zero-Order Pretraining through Model Sharding
arXiv:2609.37899v1 Announce Type: new Abstract: Zero-order optimization (ZO) trains without backpropagation, making it relevant to forward-only hardware and non-differentiable loss, but its gradient...
arXiv AI
Jul 22
GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding
arXiv:2605. 15250v3 Announce Type: replace-cross Abstract: Multi-head Latent Attention (MLA), the attention used in DeepSeek-V2/V3, jointly compresses keys and values into a low-rank latent and matches the H100 roofline almost perfectly.
arXiv Machine Learning
Jun 3
Demystifying Pipeline Parallelism: First Theory for PipeDream
arXiv:2606. 03498v1 Announce Type: new Abstract: Training modern machine learning models increasingly requires computation to be distributed across many accelerators.
arXiv Machine Learning
Sep 10
Dense Structural Compression of Transformers via Gauge-Correct Channel Removal
The paper introduces GaugeLasso, a method that applies symmetric group‑lasso penalties to transformer channels during training, enabling entire tensor slices to be zeroed out while maintaining dense tensors for GPU efficiency. By calibrating channel penalties based on inference utility per compute, the network self‑organizes into depth‑dependent structural profiles that can be dramatically smaller than the original architecture, achieving up to 255‑fold compression on a polynomial division task and outperforming hand‑designed baselines on language modeling and autoencoding benchmarks. The approach also accelerates training and reveals over‑provisioned axes that guide subsequent design iterations.