arXiv:2608. 03796v1 Announce Type: cross Abstract: Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD).
arXiv:2605.29350v2 Announce Type: replace Abstract: Sparse Mixture-of-Experts (MoE) models scale model capacity through sparse activation, but storing their full parameters remains costly in GPU memo...
arXiv:2608.20953v1 Announce Type: cross Abstract: Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and...
arXiv:2607. 13124v1 Announce Type: cross Abstract: Structured pruning is a hardware-friendly way to compress LLMs, but it is mostly validated on multiple-choice recognition tasks, while the same compressed checkpoints can collapse on the free-form generation that deployment actually requires.
AutoCompress is a hardware‑guided model compression framework that integrates latency, throughput, and memory constraints directly into differentiable architecture search. It jointly optimizes global and per‑layer structures to produce heterogeneous architectures, using serving‑engine cost models validated against real end‑to‑end measurements. In evaluations on long‑context coding, datacenter serving, and in‑game interaction, AutoCompress outperforms the Minitron pruning framework, improving average accuracy by up to 6.57 percentage points on RTX 5070 and compressing Nemotron 3.5 Super from 120B to 74B parameters while retaining 95% of its benchmark score.
arXiv:2607. 01646v2 Announce Type: replace Abstract: State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware stack.
The paper proposes a theoretical framework for scheduling high‑quality data in large language model training by extending functional scaling laws to account for time‑varying data quality. It identifies two regimes—noise‑limited and signal‑limited—where high‑quality data should be used differently, and introduces a Drop‑Stable‑Rampup training schedule that adjusts batch size at the quality transition. Experiments on 15B MoE and 600M dense models show significant accuracy gains over conventional decay schedules across multiple benchmarks.
DPA-MoE is a runtime precision control framework that allows Mixture-of-Experts (MoE) models to use any-precision routed experts. It encodes expert feed‑forward weights as prefix‑decodable chunks, profiles reconstruction residuals offline, and allocates chunks during inference based on live router gate scores. This enables a single encoded MoE checkpoint to operate at multiple effective weight budgets without retraining, improving KL divergence and downstream accuracy on Qwen3‑30B‑A3B and DeepSeekMoE‑16B with modest throughput overhead.
The paper presents a budget‑aware compression pipeline for deploying 70B‑parameter language models on a single NVIDIA GPU. It examines how pruning, quantization, and KV‑cache compression interact, showing that layer‑wise pruning improves weight quantization robustness and that KV‑cache sparsification complements INT8 KV quantization without harming decoding speed. Using these insights, the authors compressed a 70B model to ~33 GB, achieving ~57 tokens/s on 10k‑token prompts on an A40 while maintaining accuracy within 5% on standard benchmarks.
STAPLE is a method that integrates static sparse memory into pretrained language models by adding token‑indexed embeddings as a residual pathway while preserving the original feed‑forward network. On the OLMo‑2 1B and 7B models, STAPLE outperforms dense training and other static sparse baselines, improving benchmark scores by up to 4.27 points and achieving faster decoding with less GPU memory and energy usage. The approach demonstrates that large pretrained models can be expanded efficiently without costly sparse pretraining from scratch.
arXiv:2606. 18114v1 Announce Type: cross Abstract: State Space Models (SSMs) such as Mamba-2 offer linear-time inference but their memory footprint limits edge deployment.
arXiv:2606. 27866v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models.