The study investigates how to best allocate a fixed inference budget between reducing weight precision (quantization) and averaging over different answer‑option orders in small local language models. Experiments on Qwen3‑0.6B and Qwen3‑1.7B show that 6‑bit quantization retains or even slightly improves accuracy compared to fp16, while 4‑bit quantization introduces significant answer‑slot skew. Averaging four cyclic option orders can recover a large accuracy gain at 4 bits, and cost‑matched analysis indicates that using 6‑bit weights plus up to four order averages is the most efficient strategy when the budget allows at least 6 bits per weight.
By Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri
arXiv:2607. 14431v1 Announce Type: cross Abstract: We report a way to make a frozen small language model both more capable and dramatically cheaper at once, without changing any weights.
By Sietse Schelpe
The paper investigates whether disagreement between two quantized versions of the same language model can serve as an error signal for tasks such as abstention and escalation. Using Qwen3-0.6B and Qwen3-1.7B, the authors compare an 8‑bit copy with lower‑precision copies (3–6 bits) on multiple-choice datasets, measuring error detection via AUROC, risk‑coverage, and ECE. Results show modest gains over entropy alone, with additional forward passes offering limited improvement, and the study provides a deployment recipe and discusses scenarios where the method is ineffective.
By Mohammed Nayeem Khan, Allan Kevin T
arXiv:2610.05685v2 Announce Type: replace
Abstract: Reasoning models write most of their KV cache while decoding long chains of thought (CoT), so the cache has to be compressed online under a fixed m...
By Runguo Li
arXiv:2610.02598v1 Announce Type: new
Abstract: Deploying LLMs on consumer-grade GPUs with insufficient memory to hold their weights can result in prohibitively slow inference, because decoding repea...
By JuneHyung Kim, Sankeerth Durvasula, Nandita Vijaykumar
ScaleRLT introduces a hybrid learner log‑probability method that reconciles differences between dense and deferred rollout computations in looped language models like Ouro. By using a stop‑gradient replay of the rollout schedule, the approach achieves bit‑for‑bit reproducibility across FP32 and bf16 real‑weight cells while maintaining the same gradient accuracy as dense implementations. The technique boosts rollout throughput by up to 1.56× on a single GPU and significantly speeds up multi‑node GRPO updates, leading to substantial accuracy gains on GSM8K in short training runs.
By Dezhen Lu, Zixiang Ni, Runguo Li
The paper demonstrates that Mixture-of-Experts (MoE) models can avoid the need for calibration data when selecting which expert channels to load during decoding. Instead of using a calibrated threshold, a top‑k selection strategy—choosing the k highest‑scoring channels per token—achieves comparable or better performance without calibration. Additionally, the authors propose a reader that uses a ternary sketch and low‑rank mapping to recover skipped channels, achieving 98.1% of full‑model accuracy while reading only 17.8% of expert weights and speeding up inference by up to 2.69× compared to optimized vLLM.
By Geunhyoung Kim, Sung Ju Hwang
arXiv:2611.17605v1 Announce Type: cross
Abstract: A developer running a code assistant on a laptop downloads one model file, and local inference moves the energy of every request onto that machine. A...
By Aditi Patodiya
The paper investigates how various non‑quantization factors—such as engine changes, server restarts, batching differences, and kernel selection—affect the performance of quantized language models compared to their BF16 baselines. Using 2,400 public decisions from Cloudflare’s Clef‑flash (9B) and Clef (27B) models on an NVIDIA GB10, the authors quantify the noise introduced by each factor and isolate the true impact of different quantization schemes (8‑bit, FP8, Q4_K_M, NVFP4). They find that non‑quantization sources contribute up to 0.92% of decision changes, while 4‑bit quantization can alter accuracy by up to 0.95 points, and they provide code, recipes, and detailed per‑record outputs for reproducibility.
arXiv:2606. 08635v1 Announce Type: new Abstract: Prefill-decode (PD) disaggregation decouples prompt processing from token generation, but it also turns the key-value (KV) cache into a network payload.
By Yang Pengju
The paper introduces DuoRate, a method for allocating precision in Mixture-of-Experts (MoE) models that separates storage and decoding rates. It stores a uniform tier of weights once and then adapts read precision per workload using a Lagrangian rate‑distortion optimization, reducing the store‑read gap. Experiments on Qwen3-30B-A3B and GSM8K show that DuoRate achieves higher accuracy at lower bit‑per‑weight (bpw) rates compared to fixed‑format baselines.
By Mengqi Guo, Zhe Wei, Yuan Yuan, Jiunn Bin Lim, Boyi Pan, Michael Bi Mi
arXiv:2608. 08910v1 Announce Type: cross Abstract: PTQTP decomposes LLM weight matrices into two ternary (trit) planes with two free per-group scales.
By Matteo Grella