The paper introduces TTS‑Q, a perception‑guided post‑training quantization method for zero‑shot text‑to‑speech diffusion transformers. It identifies that 4‑bit weight‑and‑activation quantization erodes speaker timbre, and addresses this by selecting hard‑to‑quantize items and applying a mixed‑precision schedule that gives higher precision to late‑step feed‑forward activations. The resulting approach restores timbre while preserving intelligibility and reduces GPU memory usage for resource‑constrained deployment.
By Xiaoqin Feng, Yuqi Li, Weilun Feng, Yi-Cheng Lin, Kai Li, Xianglong Wang, Kuo Yang, Chuanguang Yang, Yingli Tian, Wei Li, Zhongwei Cheng, Lin Chen
The paper presents TTS‑Q, a post‑training quantization method that reduces the size of large diffusion transformer TTS models to 4‑bit precision while preserving speaker timbre and intelligibility. By identifying hard‑to‑quantize weights and activations and applying a mixed‑precision schedule that gives higher precision to late‑step feed‑forward activations, TTS‑Q restores timbre loss observed in standard 4‑bit PTQ. The approach also offers a calibrated‑W4A4 reference that lowers GPU memory usage, making TTS diffusion transformers more viable for on‑device, assistive, and real‑time applications.
By Xiaoqin Feng, Yuqi Li, Weilun Feng, Yi-Cheng Lin, Kai Li, Xianglong Wang, Kuo Yang, Chuanguang Yang, Yingli Tian, Wei Li, Zhongwei Cheng, Lin Chen
arXiv:2608. 08188v1 Announce Type: new Abstract: Post-training quantization reduces the deployment cost of large language models, yet how severely a quantized model degrades is not determined by bit-width alone.
By Chenxi Zhou, Pengfei Cao, Jinyu Ye, Bohan Yu, Haida Yu, Jiang Li, Jun Zhao, Kang Liu
Post-training quantization reduces the deployment cost of large language models, yet how severely a quantized model degrades is not determined by bit-width alone. We systematically study weight-only post-training quantization across bit-widths, quantization methods, model scales and downstream tasks on multiple model families.
The paper presents a new evaluation protocol for post‑training quantization of speech language models that separates lexical output, transcript‑insufficient endpoints, and packed implementations. In a Qwen2‑Audio case study, a 6‑bit allocation selected for translation improves chrF scores but degrades emotion recognition, while uniform and front‑layer controls perform better on emotion tasks. Similar patterns hold at 7 bits, and a 4‑bit study shows consistent emotion deficits across all low‑bit allocations, with no advantage for the selected scheme. The study highlights a precision‑dependent mismatch between lexical output, waveform‑dependent behavior, and nominal precision, without claiming a general failure of low‑bit models or a deployment benefit for the selected allocation.
By Mengzhe Geng, Jinxi Jin, Junhao Xu
Squeeze10-LLM is a staged mixed‑precision post‑training quantization framework that reduces 16‑bit LLM weights to an average of 1.6 bits per weight by assigning 80% of weights to 1 bit and 20% to 4 bits. It introduces Post‑Binarization Activation Robustness (PBAR), a weight significance metric that considers activation impact, and Full Information Activation Supervision (FIAS), a strategy that preserves activation information to limit error propagation. Experiments on LLaMA and LLaMA2 demonstrate that Squeeze10‑LLM achieves state‑of‑the‑art performance for sub‑2‑bit weight‑only quantization, raising average accuracy from 43% to 56% on six zero‑shot classification tasks.
By Qingcheng Zhu, Yangyang Ren, Linlin Yang, Yanjing Li, Sheng Xu, Haodong Zhu, Juan Zhang, Runqi Wang, Baochang Zhang
The paper introduces the Offline AI Modules workstream, which provides a voice‑first offline architecture, a low‑cost hardware reference bill of materials, and a reproducible quantization and benchmarking pipeline for instruction‑tuned language models in the 2‑5B parameter range. It evaluates three models across four quantization formats on two hardware tiers—NVIDIA Jetson Orin NX and Raspberry Pi5—measuring deployment metrics and multilingual quality. The key result is that Q4_K_M quantization offers the best size‑to‑quality trade‑off, enabling high decode throughput and strong topic classification accuracy while staying within memory limits on both tiers.
By Sunday Afariogun, Odunolaoluwa Jenrola, Zeinab Nezami
arXiv:2607. 07964v1 Announce Type: new Abstract: Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining.
By Donghyun Lee, Yuhang Li, Ruokai Yin, Priyadarshini Panda
The paper introduces a compression framework for the Whisper automatic speech recognition model that jointly optimizes six deployment dimensions—model size, temporal resolution, encoder token stride, low‑rank adaptation capacity, weight precision, and sparsity pattern—using NSGA‑III. The optimization targets three objectives: word error rate, inference FLOPs, and memory footprint. Evaluating 1,680 configurations, the study identifies compression combinations that outperform single‑axis scaling and notes that 1:4 structured sparsity cannot maintain acceptable accuracy within the tested budgets.
By Vyom Agarwal, Mokshda Gangrade, Siddharth Pal, Jerry Wu
arXiv:2410. 13056v4 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have demonstrated remarkable success across a wide range of language tasks, but their deployment on edge devices remains challenging due to the substantial memory requirements imposed by their large parameter sizes.
By Zihan Chen, Bike Xie, Jundong Li, Cong Shen
Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a promising compression technique, offering significant reductions in model size and inference complexity.
arXiv:2606. 13054v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment.
By Zhixiong Zhao, Zukang Xu, Zhixuan Chen, Xing Hu, Zhe Jiang, Dawei Yang