The paper introduces a composite metric for quantizing small language models that balances information retention and throughput gains. By profiling Gemma 3 1B, it identifies Feed‑Forward Network blocks and the embedding matrix as key acceleration targets, estimating quality and speed scores without actual execution. The metric, validated across architectures with ~4% prediction error, enables predictable, resource‑efficient quantization compared to evolutionary or Shapley‑based methods.
The paper argues that AI deployment performance depends on interactions among compression, compiler transformations, and serving policies rather than just model architecture. It introduces a three‑layer taxonomy—model‑level techniques, compiler transformations, and system policies—and frames deployment as a constrained multi‑objective optimization problem over accuracy, latency, throughput, memory footprint, and energy. The authors propose an evidence protocol for comparable benchmarking and synthesize data from edge and data‑center platforms to show that cross‑layer interactions drive deployment outcomes, concluding with a constraint‑aware selection procedure and open research problems.
By Tejinder Singh, John Pflueger, Jeebak Mitra, Robert Lincourt, Mitchell Markow, Bhavesh A. Patel
The paper introduces RAMP, a method for robust adaptive mixed‑precision quantization of vision models on edge CPUs. It evaluates 13 sensitivity metrics across four neural networks, finding that Jensen‑Shannon Divergence consistently identifies layers that can be safely quantized. Using K‑Means clustering on these metrics, RAMP achieves near‑lossless accuracy with an average 1.81× speed‑up, while cautioning against excluding low‑speed‑up layers that can fragment the computational graph.
By David Poblaci\'on-Criado, Dario Garcia-Gasulla, Eduardo Quinones
arXiv:2606. 07819v1 Announce Type: new Abstract: Recently, the efficiency of Large Language Models (LLMs) deployment has become a critical concern in practical applications.
By Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi, Phuong Hoai Ha
arXiv:2609.36654v1 Announce Type: new
Abstract: Large language models make weight storage and memory traffic major inference costs, motivating low-precision formats that represent each weight with on...
By Ruiyi Ding, Jie Li, Kang He, Ziyan Liu, Chengru Song, Yuedong Xu, Yuan Cheng
arXiv:2606. 04620v1 Announce Type: cross Abstract: LLMs have become the state-of-the-art algorithms for solving NLP tasks.
By Pasindu Wickramasinghe, Achyuta Muthuvelan, Rachmad Vidya Wicaksana Putra, Minghao Shao, Muhammad Shafique
arXiv:2610.07522v1 Announce Type: new
Abstract: Post-training quantization is a powerful tool for compressing large language models. The most scalable methods quantize every layer in parallel, but qu...
By Yan Scholten, Rachel Lawrence, James Hensman, Stephan G\"unnemann, Alicia Curth, Riccardo Grazzi
Deploying deep learning models on edge CPUs is bottlenecked by computational and memory constraints. Mixed-precision quantization promises to reduce inference latency while preserving accuracy. Howeve...
SpecQuant is a training‑free framework that merges speculative decoding with multi‑parent quantization to enable adaptive, efficient inference of large language models. It generates several quantized variants (INT4, FP8, FP16) from a single base model and routes queries to the appropriate variant based on predicted complexity, using lightweight models for simple tasks and full‑precision models for complex reasoning. Evaluations on Qwen2.5 models across MMLU, AlpacaEval, and GSM8K show 35‑43% speedups with less than 2% accuracy loss, facilitating practical on‑device LLM deployment without specialized infrastructure.
By Harish KB, Jagadeeswaran M, Pradheep P, Yuvanesh S, Sivakumar T
arXiv:2602. 15327v2 Announce Type: replace-cross Abstract: Machine learning model performance improvements tend to arise from competition and application.
By Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis, Sham Kakade
arXiv:2606. 04063v1 Announce Type: cross Abstract: Deploying large language models (LLMs) is challenging due to their significant memory and computational requirements.
By Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi, Phuong Hoai Ha
arXiv:2610.05265v2 Announce Type: replace
Abstract: Looped language models provide a parameter-efficient way to scale iterative test-time computation by repeatedly executing a shared recurrent core....
By Zeyu Li, Yipu Zhang, Jintao Cheng, Xin Li, Wei Zhnag