The study investigates how to best allocate a fixed inference budget between reducing weight precision (quantization) and averaging over different answer‑option orders in small local language models. Experiments on Qwen3‑0.6B and Qwen3‑1.7B show that 6‑bit quantization retains or even slightly improves accuracy compared to fp16, while 4‑bit quantization introduces significant answer‑slot skew. Averaging four cyclic option orders can recover a large accuracy gain at 4 bits, and cost‑matched analysis indicates that using 6‑bit weights plus up to four order averages is the most efficient strategy when the budget allows at least 6 bits per weight.
By Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri
arXiv:2609.37647v1 Announce Type: cross
Abstract: Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed...
By Tobias Deu{\ss}er, Lorenz Sparrenberg, Rafet Sifa
arXiv:2608. 06564v2 Announce Type: replace Abstract: Quantization is how large language models are actually deployed, and below four bits it hurts.
By Zekun Wu, Swati Dhiman, Adriano Koshiyama
The paper proposes a dynamic weight‑precision strategy for local decoding, storing 8‑bit weights but using only the top 4 bits during most steps and escalating to full precision when the low‑precision output distribution is uncertain (high entropy or small top‑two margin). Experiments on Qwen3‑0.6B with the GSM8K benchmark show that a 15% escalation budget captures 78% of low‑precision argmax flips, outperforming a random gate, and that an entropy gate at 16% escalation achieves 62.7% accuracy with only 0.87 of the weight‑byte cost of using all‑high precision. The study demonstrates that selective precision escalation can reduce computational cost while maintaining competitive performance.
By Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri
arXiv:2608. 11922v1 Announce Type: cross Abstract: Predictive-distribution entropy makes a strong selection rule in retrieval-augmented question answering: across five QA benchmarks, keeping the candidate answer that a frozen respondent LLM produces with the lowest answer-token entropy lifts mean answer $F_1$ from 0.
By Po-Jen Ko, Che-Cheng Wu, Hung-Chun Hsu, Li-Yang Chang, Chuan-Ju Wang
ThinkBit is a two‑bit scalar quantization framework designed to preserve reasoning capabilities in language models. It assigns each layer a ternary, 2‑bit, or 3‑bit tier based on sensitivity, then uses a Gumbel‑softmax search to find optimal codewords block‑by‑block. After freezing codewords, it recalibrates shared scales via knowledge‑distillation and performs a brief quantization‑aware training, finally selecting the best checkpoint with paired reasoning tests.
By Rhea Sanjay Sukthanker, Pashmina Cameron
arXiv:2609.07901v1 Announce Type: new
Abstract: Weight quantization largely determines the economics of serving open-weight LLMs. Its costs are usually assessed with capability benchmarks, on which 4...
By Dachi Kurtskhalia
arXiv:2608. 15445v1 Announce Type: new Abstract: When a reward is correct on every training example yet consistent with more than one goal, a model can acquire an unintended one, a failure known as goal misgeneralization.
By Suyash Maniyar, Armaan Sandhu, Abhishek Mishra
arXiv:2609.14825v1 Announce Type: cross
Abstract: Large language models (LLMs) are often deemed unsafe for clinical question answering because of their tendency to hallucinate. Retrieval augmentation...
By Zeyu Dong, Benjamin Wang, Joyee W. Jin
The paper presents a system for the MedReason 2026 challenge that tackles both multiple‑choice and open‑ended medical visual question answering using offline, containerized inference. Key findings include that comparing answer semantics rather than labels boosts retrieval‑only accuracy from 20.0 % to 57.5 % on a 200‑case holdout, and that varying the number of in‑prompt retrieved examples has minimal impact on final accuracy (93.5 %–94.0 %). The final system achieves 94.0 % MCQ accuracy on the development set and 93.20 % on the official pre‑evaluation, far surpassing the off‑the‑shelf baseline.
"whyItMatters":"The results demonstrate that semantic‑aware retrieval and careful adapter tuning can dramatically improve medical VQA performance, offering a practical approach for high‑accuracy, offline inference in clinical settings."
By Tristan Kirscher (ICube, Institut Strauss), Niklas C. Koser (CAU), Soren Pirk (CAU)
arXiv:2607. 27275v1 Announce Type: new Abstract: Post-training quantization to 4-bit weights is widely reported to be nearly lossless.
By Jiwon Jang, Kisu Yang, Heuiseok Lim, Hyunwoo Park
The paper introduces Neighborhood Elites (NE), a quality‑diversity search method for generating answer‑preserving rewrites in question answering. NE explicitly controls two dimensions—type of edit and explicitness of support—to create diverse perturbations that improve uncertainty quantification. Across five extractive‑QA datasets and five instruction‑tuned LLMs, NE outperforms other constructors, achieving higher AUROC and AUPRC scores and surpassing semantic entropy baselines.
By Mesut Ceylan, Arda Baris Basaran, Elgar Fleisch, Filipe Barata