arXiv AI

Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models

The study investigates how to best allocate a fixed inference budget between reducing weight precision (quantization) and averaging over different answer‑option orders in small local language models. Experiments on Qwen3‑0.6B and Qwen3‑1.7B show that 6‑bit quantization retains or even slightly improves accuracy compared to fp16, while 4‑bit quantization introduces significant answer‑slot skew. Averaging four cyclic option orders can recover a large accuracy gain at 4 bits, and cost‑matched analysis indicates that using 6‑bit weights plus up to four order averages is the most efficient strategy when the budget allows at least 6 bits per weight.

arXiv AI
3d ago

One Model, Two Precisions: Does Disagreement Between Quantized Copies Tell You When a Small Model Is Wrong?

The paper investigates whether disagreement between two quantized versions of the same language model can serve as an error signal for tasks such as abstention and escalation. Using Qwen3-0.6B and Qwen3-1.7B, the authors compare an 8‑bit copy with lower‑precision copies (3–6 bits) on multiple-choice datasets, measuring error detection via AUROC, risk‑coverage, and ECE. Results show modest gains over entropy alone, with additional forward passes offering limited improvement, and the study provides a deployment recipe and discusses scenarios where the method is ineffective.

By Mohammed Nayeem Khan, Allan Kevin T
arXiv AI
6d ago

Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model

The paper proposes a dynamic weight‑precision strategy for local decoding, storing 8‑bit weights but using only the top 4 bits during most steps and escalating to full precision when the low‑precision output distribution is uncertain (high entropy or small top‑two margin). Experiments on Qwen3‑0.6B with the GSM8K benchmark show that a 15% escalation budget captures 78% of low‑precision argmax flips, outperforming a random gate, and that an entropy gate at 16% escalation achieves 62.7% accuracy with only 0.87 of the weight‑byte cost of using all‑high precision. The study demonstrates that selective precision escalation can reduce computational cost while maintaining competitive performance.

By Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri
arXiv AI
Jul 7

HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference

arXiv:2607. 04302v1 Announce Type: cross Abstract: We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on Ascend NPUs, while maintaining the online softmax state in FP16.

By Hui Dong, Yanzhao Li, Jie Gao, Chunlu Li, Zhiyuan Zhang, Yupeng Sun, Zhenyuan Chen, Zhiqiang Zou
arXiv Computation and Language
Aug 28

Agents Don't Paginate: First-Chunk Selection for LLM Tool Responses

The paper investigates why large‑language‑model coding agents rarely request a second chunk of tool output, focusing on the precision‑at‑1 rate ($p_1$) of the gold item appearing first in the first chunk. In a benchmark of 500 software‑engineering tasks, the authors compare six value functions and find that increasing $p_1$ does not systematically improve downstream accuracy; the agent can recover the correct answer from any position within the chunk. Adding file‑metadata signals to a keyword scorer actually reduces $p_1$, while a parameter‑free keyword scorer improves $p_1$ but still fails to boost overall accuracy.

By Tatiana Petrova, Andrei Mazniak, Radu State
arXiv Machine Learning
Nov 2

bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix

The paper identifies a hidden mismatch in reinforcement‑learning fine‑tuning of large language models: when training uses fp32 master weights but rollouts are served from a bf16 copy produced by round‑to‑nearest conversion, most per‑step updates are too small to affect the served weights, causing a lag of 50–150 steps. The authors propose Fair Cast, a one‑line change that exports the master weights with an unbiased rounding rule (stochastic rounding or error feedback) and computes the loss on the served copy, eliminating the lag without adding hyperparameters and incurring only a modest computational cost. Across multiple models and optimizers, Fair Cast consistently outperforms round‑to‑nearest, improves checkpoint consistency, and reveals the implicit regularization effect of the lag.

By Andrew Gao, William Gao
Hugging Face Trending Papers
5d ago

The Noise Floor of a Quantization Comparison: Engine, restart, batch and kernel drift, measured against 8-bit and 4-bit effects in two decision models

The paper investigates how various non‑quantization factors—such as engine changes, server restarts, batching differences, and kernel selection—affect the performance of quantized language models compared to their BF16 baselines. Using 2,400 public decisions from Cloudflare’s Clef‑flash (9B) and Clef (27B) models on an NVIDIA GB10, the authors quantify the noise introduced by each factor and isolate the true impact of different quantization schemes (8‑bit, FP8, Q4_K_M, NVFP4). They find that non‑quantization sources contribute up to 0.92% of decision changes, while 4‑bit quantization can alter accuracy by up to 0.95 points, and they provide code, recipes, and detailed per‑record outputs for reproducibility.