The study investigates how to best allocate a fixed inference budget between reducing weight precision (quantization) and averaging over different answer‑option orders in small local language models. Experiments on Qwen3‑0.6B and Qwen3‑1.7B show that 6‑bit quantization retains or even slightly improves accuracy compared to fp16, while 4‑bit quantization introduces significant answer‑slot skew. Averaging four cyclic option orders can recover a large accuracy gain at 4 bits, and cost‑matched analysis indicates that using 6‑bit weights plus up to four order averages is the most efficient strategy when the budget allows at least 6 bits per weight.
By Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri
arXiv:2607. 14431v1 Announce Type: cross Abstract: We report a way to make a frozen small language model both more capable and dramatically cheaper at once, without changing any weights.
By Sietse Schelpe
The paper investigates whether disagreement between two quantized versions of the same language model can serve as an error signal for tasks such as abstention and escalation. Using Qwen3-0.6B and Qwen3-1.7B, the authors compare an 8‑bit copy with lower‑precision copies (3–6 bits) on multiple-choice datasets, measuring error detection via AUROC, risk‑coverage, and ECE. Results show modest gains over entropy alone, with additional forward passes offering limited improvement, and the study provides a deployment recipe and discusses scenarios where the method is ineffective.
By Mohammed Nayeem Khan, Allan Kevin T
arXiv:2610.05685v2 Announce Type: replace
Abstract: Reasoning models write most of their KV cache while decoding long chains of thought (CoT), so the cache has to be compressed online under a fixed m...
By Runguo Li
arXiv:2610.02598v1 Announce Type: new
Abstract: Deploying LLMs on consumer-grade GPUs with insufficient memory to hold their weights can result in prohibitively slow inference, because decoding repea...
By JuneHyung Kim, Sankeerth Durvasula, Nandita Vijaykumar
ScaleRLT introduces a hybrid learner log‑probability method that reconciles differences between dense and deferred rollout computations in looped language models like Ouro. By using a stop‑gradient replay of the rollout schedule, the approach achieves bit‑for‑bit reproducibility across FP32 and bf16 real‑weight cells while maintaining the same gradient accuracy as dense implementations. The technique boosts rollout throughput by up to 1.56× on a single GPU and significantly speeds up multi‑node GRPO updates, leading to substantial accuracy gains on GSM8K in short training runs.
By Dezhen Lu, Zixiang Ni, Runguo Li