The paper introduces a composite metric for quantizing small language models that balances information retention and throughput gains, using a normalized SQNR-based coefficient and roofline-based latency analysis. Profiling Gemma 3 1B shows that Feed‑Forward Network blocks and the embedding matrix are prime candidates for acceleration, with the metric enabling tuning of speed‑quality trade‑offs without actual execution. The authors demonstrate that their estimates predict accelerated speedup within about 4% error and allocate resources more effectively than evolutionary search or Shapley‑value methods.
By Artem Safronov
The paper argues that AI deployment performance depends on interactions among compression, compiler transformations, and serving policies rather than just model architecture. It introduces a three‑layer taxonomy—model‑level techniques, compiler transformations, and system policies—and frames deployment as a constrained multi‑objective optimization problem over accuracy, latency, throughput, memory footprint, and energy. The authors propose an evidence protocol for comparable benchmarking and synthesize data from edge and data‑center platforms to show that cross‑layer interactions drive deployment outcomes, concluding with a constraint‑aware selection procedure and open research problems.
By Tejinder Singh, John Pflueger, Jeebak Mitra, Robert Lincourt, Mitchell Markow, Bhavesh A. Patel
arXiv:2606. 07819v1 Announce Type: new Abstract: Recently, the efficiency of Large Language Models (LLMs) deployment has become a critical concern in practical applications.
By Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi, Phuong Hoai Ha
arXiv:2606. 04620v1 Announce Type: cross Abstract: LLMs have become the state-of-the-art algorithms for solving NLP tasks.
By Pasindu Wickramasinghe, Achyuta Muthuvelan, Rachmad Vidya Wicaksana Putra, Minghao Shao, Muhammad Shafique
arXiv:2610.07522v1 Announce Type: new
Abstract: Post-training quantization is a powerful tool for compressing large language models. The most scalable methods quantize every layer in parallel, but qu...
By Yan Scholten, Rachel Lawrence, James Hensman, Stephan G\"unnemann, Alicia Curth, Riccardo Grazzi
arXiv:2606. 04063v1 Announce Type: cross Abstract: Deploying large language models (LLMs) is challenging due to their significant memory and computational requirements.
By Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi, Phuong Hoai Ha
The paper introduces RAMP, a method for robust adaptive mixed‑precision quantization of vision models on edge CPUs. It evaluates 13 sensitivity metrics across four neural networks, finding that Jensen‑Shannon Divergence consistently identifies layers that can be safely quantized. Using K‑Means clustering on these metrics, RAMP achieves near‑lossless accuracy with an average 1.81× speed‑up, while cautioning against excluding low‑speed‑up layers that can fragment the computational graph.
By David Poblaci\'on-Criado, Dario Garcia-Gasulla, Eduardo Quinones
arXiv:2609.36654v1 Announce Type: new
Abstract: Large language models make weight storage and memory traffic major inference costs, motivating low-precision formats that represent each weight with on...
By Ruiyi Ding, Jie Li, Kang He, Ziyan Liu, Chengru Song, Yuedong Xu, Yuan Cheng
arXiv:2505.10202v2 Announce Type: replace
Abstract: Large Language Models (LLMs) have achieved remarkable success but face significant computational and memory challenges, particularly due to their e...
By Jintian Shao, Hongyi Huang, Jiayi Wu, YiMing Cheng, ZhiYu Wu, You Shan, MingKai Zheng
ProQuant is a weight‑only post‑training quantization framework designed for Mixture‑of‑Experts large language models. It introduces PoP, which jointly prunes experts and assigns mixed‑precision bit‑widths to better use a tight bit budget, and RoCo, a routing‑consistent RMSNorm fine‑tune that mitigates routing drift caused by quantization errors. Experiments show ProQuant outperforms existing methods on a range of generative tasks.
arXiv:2607. 22583v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved widespread adoption because of their strong reasoning and query-response capabilities.
By Muhammad Junaid Ali, Smail Niar, El-Ghazali Talbi
arXiv:2607. 20806v1 Announce Type: new Abstract: Lightweight large language models (LLMs) are increasingly being deployed locally on personal computers and are expected to play a growing role in resource-constrained edge and mobile environments.
By Tomohiro Harada, Enrique Alba, Gabriel Luque