arXiv:2602.14635v2 Announce Type: replace-cross Abstract: Model compression is necessary for deployment of transformer models in resource-constrained environments. However, compression reduces the pe...
arXiv:2607.29397v3 Announce Type: replace Abstract: Deploying large language models in realistic server environments poses challenges, as the system needs to provide high-quality responses with low l...
The paper introduces Trainable Vector Quantization (TVQ), a method that uses straight-through estimation to allow gradient descent optimization of vector-quantized large language models. TVQ reformulates VQ with trainable latent weights, enabling arbitrary objective minimization and is applied in a block-wise knowledge distillation framework to balance quality and training efficiency. Experiments show TVQ achieves state‑of‑the‑art low‑bit post‑training quantization, achieving near‑lossless 2‑bit quantization for LLaMA2‑70B with only a 1% accuracy drop on question‑answering benchmarks.
arXiv:2603. 01875v3 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is an essential technique to compress large language models (LLMs) into smaller ones.
The paper presents a comprehensive evaluation of post‑training quantization (PTQ) techniques for large language models (LLMs), addressing gaps in prior work by conducting fair, large‑scale experiments on text‑only LLMs. It decouples quantization into pre‑quantization transformation and error mitigation, systematically studies granularity, symmetry, and data formats (MXFP4, NVFP4), and finds that rotation‑based transformations and low‑rank compensation with GPTQ can improve performance, while finer granularity and asymmetric quantization benefit activations more than weights. The study highlights the importance of understanding component interactions and the impact of scaling factor formats on low‑bit quantization performance.
arXiv:2602.09691v2 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is a technique to compress a larger teacher system into a smaller student. In machine translation, KD is commonly...
The paper reports six submissions by the ESTS team to the WMT26 Model Compression Shared Task for English–Simplified Chinese and English–Egyptian Arabic. Each submission offers three compression operating points derived from GPT‑OSS‑20B, using routing‑informed expert pruning, cross‑lingual routing divergence for capacity allocation, and MXFP4 quantization of retained expert projection weights. The resulting models, ranging from 4.186 B to 7.770 B parameters, are fine‑tuned on GPT‑5.1 synthetic data and evaluated internally with xCOMET‑XL.
DictLLM is a post‑training compression method that reduces the memory footprint of large language models by decomposing weight matrices into shared dictionaries and sparse layer‑specific coefficients. The approach involves four steps: extracting shared dictionaries via SVD, sparsifying coefficients with a dictionary‑aware Hessian, refining dictionaries, and aggressively quantizing both components using Hessian information. Experiments show that DictLLM outperforms existing low‑rank compression techniques, achieving a 5.1× compression of LLaMA‑7B with only a 2.9‑point increase in WikiText‑2 perplexity, and a 3.6× higher compression ratio than the state‑of‑the‑art method.
arXiv:2601. 22709v5 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss.
arXiv:2606. 07819v1 Announce Type: new Abstract: Recently, the efficiency of Large Language Models (LLMs) deployment has become a critical concern in practical applications.
arXiv:2605.28042v2 Announce Type: replace-cross Abstract: Modern large language models (LLMs) achieve state-of-the-art machine translation performance, but they do so as broad generalists largely tra...
arXiv:2609.36965v1 Announce Type: new Abstract: System One models such as Jev offer an efficient alternative to generative language models for tasks that require decisions rather than open-ended resp...