arXiv AI
SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Jun 22
Hybrid Compression: Integrating Pruning and Quantization for Optimized Neural Networks
Deep neural networks have witnessed remarkable advancements in recent years and have become integral to various applications. However, alongside these developments, training and deployment of neural network models on embedding and edge devices face significant challenges due to limited memory and computational resources.
arXiv Machine Learning
1d ago
Trainable Vector Quantization for Large Language Models
The paper introduces Trainable Vector Quantization (TVQ), a method that uses straight-through estimation to allow gradient descent optimization of vector-quantized large language models. TVQ reformulates VQ with trainable latent weights, enabling arbitrary objective minimization and is applied in a block-wise knowledge distillation framework to balance quality and training efficiency. Experiments show TVQ achieves state‑of‑the‑art low‑bit post‑training quantization, achieving near‑lossless 2‑bit quantization for LLaMA2‑70B with only a 1% accuracy drop on question‑answering benchmarks.
arXiv AI
Oct 9
When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization
The paper examines weight-only post‑training quantization (PTQ) and finds that minimizing reconstruction loss does not always improve model performance, sometimes even harming it on calibration data. It shows that weights with low reconstruction loss on one distribution can perform worse when activation distributions shift. To address this, the authors introduce Distributionally Robust Quantization (DRQ), a refinement step that minimizes worst‑case reconstruction loss over a constrained set of activation distributions while keeping the quantization grid unchanged. Experiments demonstrate that DRQ improves models quantized by several PTQ methods across dense and mixture‑of‑experts large language models, offering better downstream performance without extra inference cost.
arXiv AI
Jun 4
LLM Compression with Jointly Optimizing Architectural and Quantization choices
arXiv:2606. 04063v1 Announce Type: cross Abstract: Deploying large language models (LLMs) is challenging due to their significant memory and computational requirements.
arXiv AI
Jun 8
Scale When Needed: Adaptive Neuron-level Mixed Precision Quantization Aware Training
arXiv:2605. 25054v2 Announce Type: replace-cross Abstract: Deploying deep neural networks on resource-constrained 6G edge devices demands aggressive compression with minimal accuracy loss.
arXiv AI
Jun 9
Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression
arXiv:2606. 07819v1 Announce Type: new Abstract: Recently, the efficiency of Large Language Models (LLMs) deployment has become a critical concern in practical applications.