arXiv Machine Learning
Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 5
Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
arXiv:2608. 03796v1 Announce Type: cross Abstract: Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD).
arXiv AI
Oct 15
ConMoE: MoE Compression as Expert Selection and Functional Assignment
arXiv:2605.29350v2 Announce Type: replace Abstract: Sparse Mixture-of-Experts (MoE) models scale model capacity through sparse activation, but storing their full parameters remains costly in GPU memo...
arXiv AI
Aug 24
Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs
arXiv:2608.20953v1 Announce Type: cross Abstract: Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and...
arXiv AI
Jul 16
ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
arXiv:2607. 13124v1 Announce Type: cross Abstract: Structured pruning is a hardware-friendly way to compress LLMs, but it is mostly validated on multiple-choice recognition tasks, while the same compressed checkpoints can collapse on the free-form generation that deployment actually requires.
arXiv AI
6d ago
AutoCompress: Automated Hardware-Guided Model Compression
AutoCompress is a hardware‑guided model compression framework that integrates latency, throughput, and memory constraints directly into differentiable architecture search. It jointly optimizes global and per‑layer structures to produce heterogeneous architectures, using serving‑engine cost models validated against real end‑to‑end measurements. In evaluations on long‑context coding, datacenter serving, and in‑game interaction, AutoCompress outperforms the Minitron pruning framework, improving average accuracy by up to 6.57 percentage points on RTX 5070 and compressing Nemotron 3.5 Super from 120B to 74B parameters while retaining 95% of its benchmark score.
arXiv Machine Learning
Jul 8
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
arXiv:2607. 01646v2 Announce Type: replace Abstract: State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware stack.