arXiv Machine Learning September 21, 2026 By Harish KB, Jagadeeswaran M, Pradheep P, Yuvanesh S, Sivakumar T

SpecQuant: Speculative Decoding with Multi-Parent Quantization for Adaptive LLM Inference

Read the original on arXiv Machine Learning →

SpecQuant is a training‑free framework that merges speculative decoding with multi‑parent quantization to enable adaptive, efficient inference of large language models. It generates several quantized variants (INT4, FP8, FP16) from a single base model and routes queries to the appropriate variant based on predicted complexity, using lightweight models for simple tasks and full‑precision models for complex reasoning. Evaluations on Qwen2.5 models across MMLU, AlpacaEval, and GSM8K show 35‑43% speedups with less than 2% accuracy loss, facilitating practical on‑device LLM deployment without specialized infrastructure.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 7

AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference

arXiv:2607. 03876v1 Announce Type: new Abstract: With the rise of small quantized GGUF-based language models and their increasing use for on-device inference tasks, we have seen the growing need for an approach capable of reliably delivering these models at scale even under severe memory bandwidth constraints such as those imposed by pure CPU implementations.

By Sadra Saremi

llms agents reinforcement-learning efficiency

arXiv Machine Learning

Jul 3

QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs

arXiv:2602. 10431v4 Announce Type: replace Abstract: Large language models (LLMs) demand substantial computational and memory resources, posing challenges for efficient deployment.

By Kanghyun Noh, Jinheon Choi, Yulhwa Kim

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jul 3

Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

arXiv:2607. 01831v1 Announce Type: cross Abstract: Long-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems.

By Wenchen Han, Gingfung Matthew Yeung, Marco Barletta, William Toner, Amory Hoste, Adam Barker

llms rag agents efficiency benchmarks

arXiv Machine Learning

Jul 7

Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B

arXiv:2607. 04244v1 Announce Type: new Abstract: This report describes our approach to the Efficient Qwen Competition, where the goal is to enable low-latency serving of Qwen3.

By Jaeyeon Kim, Jewon Lee, Bo-Kyeong Kim

llms diffusion efficiency

Hugging Face Trending Papers

Aug 25

Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation

Post-training quantization lowers the memory footprint of Large Language Models (LLMs) and speeds up inference, which is why it is now common for on-device deployment. Most of what we know about its e...

llms efficiency benchmarks

arXiv Machine Learning

Jun 8

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization

arXiv:2605. 08692v2 Announce Type: replace Abstract: Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference.

By Beshr IslamBouli, David Jin

llms efficiency

One email a morning, machine-written

Related stories

AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference

QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs

Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B

Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization