arXiv Machine Learning

Affinity-Aware Sharding for Delayed Tensor Parallelism

arXiv AI
Jun 9

Scaling Neural Network Verification with Tensor Parallelism and Fully Sharded Data Parallelism

arXiv:2606. 09377v1 Announce Type: cross Abstract: Formal neural network verification -- proving that a network satisfies safety properties for \emph{all} inputs in a specified domain -- is bounded in practice by GPU memory: standard implementations of bound-propagation algorithms (IBP, CROWN, $\alpha$-CROWN) require weight and relaxation-coefficient matrices to reside entirely on one accelerator.

By Sergei Vorobyov, Eugene Ilyushin
Hugging Face Trending Papers
Jun 8

Scaling Neural Network Verification with Tensor Parallelism and Fully Sharded Data Parallelism

Formal neural network verification -- proving that a network satisfies safety properties for \emph{all} inputs in a specified domain -- is bounded in practice by GPU memory: standard implementations of bound-propagation algorithms (IBP, CROWN, $α$-CROWN) require weight and relaxation-coefficient matrices to reside entirely on one accelerator. We adapt two parallelism techniques originally developed for large-scale model training to the \texttt{auto\_LiRPA}\,/\,$α,β$-CROWN verification framework.

arXiv Machine Learning
Sep 30

Scaling Zero-Order Pretraining through Model Sharding

arXiv:2609.37899v1 Announce Type: new Abstract: Zero-order optimization (ZO) trains without backpropagation, making it relevant to forward-only hardware and non-differentiable loss, but its gradient...

By Francois Chaubard, Mykel J. Kochenderfer, Chris R\'e
arXiv Machine Learning
Oct 28

BellowsKV: One Latent for Many Layers, Expanded on Demand

BellowsKV introduces a training‑free approach that maps the hidden states of multiple adjacent layers in a large language model to a single shared latent representation per token, allowing each layer to retrieve its own keys and values via a layer‑specific projection. The method retains the latent at full dimension and offers two compression strategies: BellowsKV‑Quant, which stores the latent at low precision, and BellowsKV‑Evict, which makes a shared eviction decision per block. Experiments on Llama‑3.1‑8B, Mistral‑7B, and Qwen2.5‑7B show that BellowsKV‑Quant preserves about 99% of full‑cache accuracy at 3.6× compression and 97‑98% at 13‑14×, outperforming existing cross‑layer methods that drop to at most 84% accuracy at 16× compression.

By Jiamu Zhang, Hanjie Chen
arXiv Machine Learning
6d ago

Decode Token Parallelism: Memory-Efficient Distributed Attention Parallelism for Scalable LLM Decoding

The paper introduces Decode Token Parallelism (TKNP), a new decoding architecture that separates model weight placement from attention computation. TKNP distributes a single model copy across nodes in balanced layer blocks, allowing each node to compute attention for its own requests using local KV caches while performing dense operations within a single NVLink domain. Implemented in vLLM, TKNP achieves up to 1.7× higher throughput than decode context parallelism and up to 2.06× over data parallelism on large models such as Llama 3.1 405B and Nemotron‑4 340B across 64 H100 GPUs.

By Susav Shrestha, Brad Settlemyer, Narasimha Reddy
arXiv Machine Learning
Sep 10

Dense Structural Compression of Transformers via Gauge-Correct Channel Removal

The paper introduces GaugeLasso, a method that applies symmetric group‑lasso penalties to transformer channels during training, enabling entire tensor slices to be zeroed out while maintaining dense tensors for GPU efficiency. By calibrating channel penalties based on inference utility per compute, the network self‑organizes into depth‑dependent structural profiles that can be dramatically smaller than the original architecture, achieving up to 255‑fold compression on a polynomial division task and outperforming hand‑designed baselines on language modeling and autoencoding benchmarks. The approach also accelerates training and reveals over‑provisioned axes that guide subsequent design iterations.

By Jed A. Duersch, Na\"im Es-Sebbani, Nathana\"el Haas, Zied Bouraoui
arXiv Machine Learning
Oct 15

GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding

The paper introduces Group-Query Latent Attention (GQLA), a variant of Multi-head Latent Attention that shares group‑sharded up‑projections to enable a joint latent representation and a shardable expanded cache. GQLA supports both MQA‑absorb and GQA execution using the same weights and existing kernels, and roofline analysis shows how absorption trades traffic for computation while GQA shards KV groups. Experiments on DeepSeek‑V3.1‑Base and other models demonstrate that GQLA achieves comparable training losses to MLA, maintains high accuracy after checkpoint conversion, and improves output throughput by up to 35% on eight H20‑3e GPUs, while also allowing larger batch sizes due to a smaller per‑rank cache.

By Fanxu Meng, Zhixin Pan