arXiv Machine Learning By Kuanlin Chen, Chen-Wei Kuo, Cheng-En Ou

Global Ranks Survive, Selected Heads Shift: BOS-Sink Topology under 4-bit Weight-Only Quantization

Read the original on arXiv Machine Learning →

The paper investigates whether sink-aware attention head selection remains valid after 4‑bit NF4 weight‑only post‑training quantization. Using Sink Topology Consistency metrics, it finds that global rank preservation stays high across Qwen2.5 and Llama‑3.2 models, yet top‑k head overlap drops to 61–79% and layer‑specific sink‑mass shifts can be substantial. The study also shows that cross‑domain calibration degrades more than within‑domain precision and that recalibration with a small number of samples can recover most of the stability, though full‑map stability may require updating more layers.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Sep 3

Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment

The paper reports a post‑training ternarization of the 4‑billion‑parameter Qwen model, achieving an effective 1.641‑bit representation for 81.62 % of its weights while keeping activations at 16‑bit precision. Accuracy drops from 64.5 % to 54.7 % across ten capability tests, with uneven degradation (e.g., BoolQ 84.6 % of teacher performance, ARC‑Challenge 43.8 %). After packing the ternary planes, the model size shrinks from 8.29 GiB to 3.96 GiB with negligible change in perplexity, though inference speed is not improved.

By Anirudh Malik, M Sparsh Mehra, Poojith Devan
arXiv AI
Oct 1

ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights

ShamAN-Q is a sub‑1‑bit post‑training quantization technique that builds on NanoQuant by replacing its diagonal reconstruction geometry with a dense curvature metric inspired by the Shampoo optimizer. For each linear weight, it fits a Kronecker product to the empirical Fisher information matrix of a small calibration set via Kullback–Leibler minimization, yielding a Mahalanobis reconstruction loss. The method updates continuous ADMM steps to Sylvester equations while keeping the discrete projection and deployment format unchanged, and it redistributes uniform rank across layers, achieving lower perplexity on Qwen3‑Base at roughly 1 bpw and matching or improving zero‑shot accuracy on the Eleuther LM Evaluation Harness.

By Jonathan Mei, Sang Hyub Kim, Oliver Knitter, Chi Chen, Martin Roetteler