arXiv AI By Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri

Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models

Read the original on arXiv AI →

The study investigates how to best allocate a fixed inference budget between reducing weight precision (quantization) and averaging over different answer‑option orders in small local language models. Experiments on Qwen3‑0.6B and Qwen3‑1.7B show that 6‑bit quantization retains or even slightly improves accuracy compared to fp16, while 4‑bit quantization introduces significant answer‑slot skew. Averaging four cyclic option orders can recover a large accuracy gain at 4 bits, and cost‑matched analysis indicates that using 6‑bit weights plus up to four order averages is the most efficient strategy when the budget allows at least 6 bits per weight.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
3d ago

One Model, Two Precisions: Does Disagreement Between Quantized Copies Tell You When a Small Model Is Wrong?

The paper investigates whether disagreement between two quantized versions of the same language model can serve as an error signal for tasks such as abstention and escalation. Using Qwen3-0.6B and Qwen3-1.7B, the authors compare an 8‑bit copy with lower‑precision copies (3–6 bits) on multiple-choice datasets, measuring error detection via AUROC, risk‑coverage, and ECE. Results show modest gains over entropy alone, with additional forward passes offering limited improvement, and the study provides a deployment recipe and discusses scenarios where the method is ineffective.

By Mohammed Nayeem Khan, Allan Kevin T
arXiv AI
6d ago

Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model

The paper proposes a dynamic weight‑precision strategy for local decoding, storing 8‑bit weights but using only the top 4 bits during most steps and escalating to full precision when the low‑precision output distribution is uncertain (high entropy or small top‑two margin). Experiments on Qwen3‑0.6B with the GSM8K benchmark show that a 15% escalation budget captures 78% of low‑precision argmax flips, outperforming a random gate, and that an entropy gate at 16% escalation achieves 62.7% accuracy with only 0.87 of the weight‑byte cost of using all‑high precision. The study demonstrates that selective precision escalation can reduce computational cost while maintaining competitive performance.

By Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri