arXiv Machine Learning

Saturation Makes Quantization Error Additive: A Coverage Model with a Certificate

arXiv:2607. 12266v1 Announce Type: new Abstract: Mixed-precision quantization must decide which parts of a model to keep at higher precision.

arXiv Machine Learning

Jul 31

Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

arXiv:2607. 27275v1 Announce Type: new Abstract: Post-training quantization to 4-bit weights is widely reported to be nearly lossless.

By Jiwon Jang, Kisu Yang, Heuiseok Lim, Hyunwoo Park

llms agents efficiency benchmarks safety

arXiv Machine Learning

Jul 7

Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models

arXiv:2607. 02893v1 Announce Type: new Abstract: Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width.

By Hamish Ogilvy

llms efficiency

arXiv Machine Learning

Aug 10

Quantization Damage Is Multiplicative, Not Additive

arXiv:2608. 06564v1 Announce Type: new Abstract: Quantization is how large language models are actually deployed, and below four bits it is known to hurt.

By Zekun Wu, Swati Dhiman, Adriano Koshiyama

llms agents efficiency benchmarks safety

arXiv Machine Learning

6d ago

SoftWater: Class-Aware Rate Allocation for Softmax Quantization

arXiv:2608. 12026v1 Announce Type: new Abstract: Post-training quantization pipelines routinely leave the softmax output layer in high precision.

By Joao V. Cavalcanti, Ashia C. Wilson

llms efficiency

arXiv AI

Jul 28

MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models

arXiv:2607. 23047v1 Announce Type: cross Abstract: Mixed-precision quantization improves the accuracy of post-training quantization by allocating higher bitwidths to sensitive layers, but existing methods solve the allocation for a single fixed memory budget.

By Ashitabh Misra, Madhav Agrawal, Arham Jain, Tarek Abdelzaher

llms efficiency

arXiv Machine Learning

Jul 14

RDQ: Residual Distribution Quantization for Large Language Models

arXiv:2607. 10137v1 Announce Type: new Abstract: Post-training quantization (PTQ) of large language models degrades sharply below 4-bit precision.

By Prateek Singh

llms efficiency benchmarks

arXiv AI

Jun 4

Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data

arXiv:2606. 04238v1 Announce Type: cross Abstract: Aggressive weight quantization to 2-bit precision offers substantial throughput and memory gains for large language model (LLM) inference, but typically incurs severe accuracy degradation.

By Devleena Das, Rajeev Patwari, Elliott Delaye, Ashish Sirasao

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jul 21

Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding

arXiv:2607. 16721v1 Announce Type: new Abstract: The strongest open-weight coding models are mixture-of-experts (MoE) networks: most of their size comes from large pools of "expert" subnetworks, of which only a few act on any token.

By Anik Jha

llms agents fine-tuning efficiency benchmarks

arXiv AI

Aug 3

WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

arXiv:2607. 28699v1 Announce Type: cross Abstract: KV-cache quantization is validated today by offline benchmark averages; a deployed system cannot tell whether compression is damaging the request it is serving right now.

By Fanzhe Wei, Li Liu

efficiency benchmarks

arXiv Machine Learning

Aug 11

Which Decisions Low-Bit Quantization Breaks, and How to Predict Them

arXiv:2608. 06564v2 Announce Type: replace Abstract: Quantization is how large language models are actually deployed, and below four bits it hurts.

By Zekun Wu, Swati Dhiman, Adriano Koshiyama

llms agents efficiency benchmarks safety

arXiv Machine Learning

Jul 7

LGQ: Learnable Geometric Quantization for Image Tokenization

arXiv:2602. 16086v3 Announce Type: replace-cross Abstract: Recent collapse-free quantizers such as FSQ achieve stable training by replacing the learnable codebook with an engineered geometry: a fixed scalar grid whose structure is dictated by the codebook size K.

By Idil Bilge Altun, Mert Onur Cakiroglu, Elham Buxton, Mehmet Dalkilic, Hasan Kurban

diffusion nlp efficiency benchmarks

arXiv AI

Jun 10

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

arXiv:2606. 09864v1 Announce Type: cross Abstract: Key-value (KV) cache quantization is widely used to reduce Large Language Model (LLM) inference memory, yet existing evaluations solely focus on measuring perplexity and accuracy without assessing the safety impact.

By Bruce Changlong Xu, Adarsh Kumarappan, Mu Zhou

llms efficiency benchmarks safety

Related stories