The paper investigates post‑training quantization of transformer attention blocks by optimizing a joint loss over the Q, K, V projections rather than individual weight matrices. Using this joint attention‑based objective (JAB), the authors achieve significant compression on Mistral‑7B, recovering 77‑90% of the performance gap at 3 bits, but the method fails when MLP layers are included. A role‑aware offset rule that ignores sensitivity estimates outperforms JAB on GPT‑2 and full Mistral‑7B, demonstrating that the matrix a weight belongs to is more critical than sensitivity metrics.
The paper introduces a label‑free mixed‑precision post‑training quantization technique for text embedders called Quantize by Drift. It uses quantization‑induced representation drift—measured by re‑encoding a corpus after quantizing a module—to estimate module sensitivity without relevance labels. Across five development embedders, drift‑based ordering of mixed‑precision plans correlates strongly with held‑out retrieval quality (macro Spearman 0.911) and outperforms uniform precision under a hard byte budget, though it does not fully replace fine‑grained redistribution strategies.
By Hyojung Han, Jongmin Kim, Seung-Hun Jeon
MRVQ (Matryoshka Residual Vector Quantization) is a single post‑hoc residual quantizer designed for frozen embeddings that can be truncated to support any combination of embedding dimension and code rate. By dropping residual stages or embedding coordinates, MRVQ provides a single resident artifact that serves all evaluated (dimension, rate) pairs, achieving the lowest memory usage among tested designs. Although it uses 17.8–22.0× less memory than three separately trained QINCo2 indices and 1.89–2.02× less than a lean shared‑model steelman, it trades off a small drop in nDCG@10 on FiQA compared to per‑rate QINCo2, yet still outperforms PQ, OPQ, and AdANNS‑OPQ at matched code size.
By Sean Culatana, Shang-En Huang, Kang Li
The paper investigates when post‑training quantization (PTQ) calibration helps or hurts text embedders, focusing on the amount of calibration (dose) and whether to calibrate at all. Across three models and 161 calibration runs at 4‑bit weight quantization, the effect ranges from significant gains to catastrophic degradation, driven mainly by the functional displacement of the quantized model. A validation‑gated abstention rule is proposed that avoids harmful calibrations while capturing most benefits, highlighting the trade‑off between safety and opportunity.
By Hyojung Han, Ryangkyung Kang, Youngeun Kim, Seung-Hun Jeon
arXiv:2608.20873v2 Announce Type: new
Abstract: A deployed language model is a file that many things depend on - a benchmark report, a certification, a fleet of devices - and every way of teaching it...
By Zifeng Liu, Yaxin Lu, Xuanhan Wu, Zhiyong Du, Yiming Mao, Zhenhe Wang, Wenqi Shi, Zhengkun Jing, Linwei Liu
arXiv:2608.20953v1 Announce Type: cross
Abstract: Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and...
By Bakbergen Ryskulov, Iker Garc\'ia-Ferrero, David Montero, David Jansen, Ali Hashemi, Jezabel R. Garcia, Antonio Tiene, Rom\'an Or\'us
The paper investigates where post‑training quantization (PTQ) harms large language models (LLMs) and how to best allocate a limited precision budget. By causally raising each layer to 8‑bit precision across nine open‑weight models, the authors find that quantization damage is diffuse rather than concentrated in specific task circuits or weight statistics, and that globally refining quantization granularity outperforms selectively protecting the most recoverable layers. They also observe that the residual accuracy loss is budget‑limited and that peak recovery locations correlate with architecture within families but not across families.
By Jundong Hu, Shekar Ramachandran
The paper reports a large‑scale post‑training ternarisation of the Qwen3 language model, extending a conversion pipeline from the 4B to the 8B variant. Using KOTMS rotation, E2M‑ATQ adaptive ternarisation, and GPTQ‑style error compensation, the authors achieve a 1.361× perplexity ratio across three corpora and retain 78.5% of the FP16 accuracy on zero‑shot tasks, with the 8B model outperforming the 4B by 8.9 percentage points. The study also demonstrates lossless lattice‑aware packing, producing an 8.24 GiB checkpoint that preserves perplexity, and shows that direct packed execution can reach 15.52 tokens/s in 7.35 GiB, though packed GEMV remains slower than FP16 cuBLAS.
By Anirudh Malik, M Sparsh Mehra, Poojith Devan
arXiv:2610.28537v1 Announce Type: new
Abstract: Post-training quantization (PTQ) compresses the weights of a trained model with limited compute and data. Down to 4 bits it is near-lossless, but at 2...
By Rhea Sanjay Sukthanker, Riccardo Grazzi, James Hensman, Pashmina Cameron
arXiv:2608.20873v1 Announce Type: new
Abstract: Every way of teaching a deployed language model something new -- full fine-tuning, adapter merging, model editing -- replaces the released checkpoint,...
By Zifeng Liu, Zhiyong Du, Yaxin Lu, Yiming Mao, Zhenhe Wang, Wenqi Shi, Zhengkun Jing
The paper investigates a failure mode in Graph-JEPA, a joint‑embedding predictive model trained on a large scientific‑reasoning graph. Despite achieving high linear‑probe accuracy and effective rank, the learned representation contains almost no usable instance information, as shown by retrieval metrics. The authors diagnose the issue to variance allocation in the objective, propose a repair that restores near‑perfect information recovery, and demonstrate that the problem persists even after repair, highlighting limitations in the evaluation metrics used.
By Gollam Rabby, S\"oren Auer
arXiv:2609.26147v1 Announce Type: new
Abstract: Modern LLMs are deployed as families of post-trained variants (base, instruct, chat, code) derived from a shared set of pre-trained weights. We present...
By Zhaohui Wang