arXiv Machine Learning
Jun 17
Price of metric universality in vector quantization is at most 0.11 bit
arXiv:2602. 05790v2 Announce Type: replace-cross Abstract: Fast computation of a matrix product $W^\top X$ is a workhorse of modern LLMs.
Minimum Distortion Quantization with Specified Output Distribution
arXiv:2606. 10458v1 Announce Type: cross Abstract: We derive the optimal quantizer of a real-valued random variable $W$ with distribution $P_W$ such that 1) the distribution of the quantization output $X$ that can take $k$ values follows any specified distribution $P_X$ over $\{1,\ldots,k\}$, and 2) the minimum mean squared error (MMSE) of estimating $W$ from $X$ is minimized.
Related stories
arXiv AI
Jun 9
High-Rate Quantized Matrix Multiplication II
arXiv:2605. 13768v2 Announce Type: replace-cross Abstract: This is the second part of the work investigating quantized matrix multiplication (MatMul).
arXiv Machine Learning
Sep 24
Exact Minimax One-Bit Unbiased Compression: Heavy-Tail Necessity and Finite-Randomness Approximation
arXiv:2609. 27860v1 Announce Type: new Abstract: A pointwise-unbiased one-bit compressor reconstructs every real input in expectation while transmitting one bit.
arXiv Machine Learning
4d ago
Mutual Information Constrained Chernoff Bottleneck
arXiv:2609.37994v1 Announce Type: cross Abstract: The classical information bottleneck (IB) measures the relevance of a representation $U$ of $X$ to a target $Y$ by $I(U;Y)$, which does not directly...
Hugging Face Trending Papers
Sep 21
Adversarially Robust PAC Learning with Optimal VC Rates
We study the problem of \emph{adversarially robust} PAC learning. In this framework, the learner observes independent samples from an unknown distribution over $\mathcal{X} \times \{0,1\}$, as in clas...
arXiv Machine Learning
Sep 10
A Note on Scaling in Randomly Rotated Quantization and Its Connection to the CDEF +1 Pythagorean Relation
arXiv:2609.08759v1 Announce Type: cross Abstract: Quantization schemes based on randomized rotations have recently received renewed attention, including the roles of MMSE and unbiased reconstruction...
arXiv Machine Learning
Aug 27
Transforms for LLM Quantization: The Great Inversion and Format Co-Design
The paper surveys the use of linear, function‑preserving transforms in 4‑bit large‑language‑model (LLM) quantization, formalizing the underlying principle as the "Great Inversion"—the trade‑off between energy concentration favored by allocation‑flexible coding and within‑group flattening favored by grouped shared‑scale quantization. It reviews 200 works, classifies 43 transform methods by structure, data‑awareness, construction approach, and runtime cost, and examines how they interact with GPTQ rounding. The study also explores how different number formats (FP4, MXFP4, NVFP4) influence the optimal transform choice and outlines open research problems. "whyItMatters":"The survey clarifies the conflicting objectives in transform‑based LLM quantization and provides a practical guide for selecting transforms based on deployment regime, thereby informing future research and deployment strategies."
arXiv Machine Learning
5d ago
Generalization behavior of OPTQ and the role of regularization
The paper investigates the generalization behavior of the OPTQ quantization algorithm and its stochastic variant. It derives bounds on the expected squared error when a test point is drawn from a fixed distribution, linking this error to the calibration dataset and to the regularization parameter λ. The authors use these theoretical insights to propose a new recommendation for choosing λ, which shows improved performance in experiments compared to previous suggestions.
arXiv Machine Learning
Jun 2
Robust Learning of a Group DRO Neuron
arXiv:2601. 18115v2 Announce Type: replace Abstract: We study the problem of learning a single neuron under standard squared loss in the presence of arbitrary label noise and group-level distributional shifts, for a broad family of covariate distributions.
arXiv Machine Learning
Sep 10
A Nuclear-Norm Lower Bound for Dithered Scalar Quantization of Matrix Products
arXiv:2609.05641v1 Announce Type: cross Abstract: We consider the problem of minimizing error in quantized matrix multiplication $C=AB$. Scalar quantization of the factors introduces rounding errors...
arXiv Machine Learning
Sep 22
Adversarially Robust PAC Learning with Optimal VC Rates
arXiv:2609.24260v1 Announce Type: cross Abstract: We study the problem of \emph{adversarially robust} PAC learning. In this framework, the learner observes independent samples from an unknown distrib...
arXiv Machine Learning
Jun 30
Universality of empirical risk minimization
arXiv:2202. 08832v3 Announce Type: replace-cross Abstract: We study a general class of optimization problems with decision variable $\boldsymbol{\Theta} \in \mathbb{R}^{p \times k}$ and cost function which is the sum of $n$ terms, each dependent on $\boldsymbol{\Theta}$ through the $k$-dimensional projection $\boldsymbol{\Theta}^\top \boldsymbol{x}_i$, where $\boldsymbol{x}_i$, $i \leq n$ are i.