arXiv AI
Jun 9
High-Rate Quantized Matrix Multiplication II
arXiv:2605. 13768v2 Announce Type: replace-cross Abstract: This is the second part of the work investigating quantized matrix multiplication (MatMul).
A Note on Scaling in Randomly Rotated Quantization and Its Connection to the CDEF +1 Pythagorean Relation
Related stories
arXiv AI
1d ago
TORQUE: Optimizing What (not) to Quantize Before and After Rotation
The paper introduces TORQUE, a framework that enhances quantization by jointly optimizing which coordinates to keep at high precision before and after applying uniform random rotations, all within a fixed bit budget. By preserving large input coordinates before rotation and the largest-magnitude coordinates after rotation, TORQUE reduces quantization error and allows efficient use of offline-optimized codebooks. The authors provide an error upper bound, prove that top‑k pre‑rotation retention is optimal for each k, and demonstrate improved accuracy‑storage tradeoffs in Gaussian models and practical tasks such as nearest‑neighbor retrieval, KV‑cache compression, and activation compression.
arXiv Machine Learning
Sep 3
Exact Limits of Random Projections for Preserving Geometry: Distance Recovery, Nearest-Neighbor Rankings, and Covariance Shape in Gaussian Models
arXiv:2609. 02155v1 Announce Type: new Abstract: The Johnson-Lindenstrauss (JL) lemma guarantees that a random projection of $n$ points to $m=O(\varepsilon^{-2}\log n)$ dimensions preserves pairwise squared distances within relative error $\varepsilon$ with high probability, and this dimension order is asymptotically optimal.
arXiv Machine Learning
Jun 11
Bernstein-Schur Kernels: Random Features by Sketched Modulation and Radial Randomization
arXiv:2606. 11255v1 Announce Type: new Abstract: Bernstein--Schur kernels are products of a finite-feature kernel (one with an explicit finite-dimensional feature map) and a completely monotone shift-invariant kernel: nonstationary kernels that fall between the shift-invariant and dot-product templates random features usually exploit, so in general neither Bochner sampling nor polynomial sketching applies to the full kernel directly.
arXiv AI
Jun 10
Minimum Distortion Quantization with Specified Output Distribution
arXiv:2606. 10458v1 Announce Type: cross Abstract: We derive the optimal quantizer of a real-valued random variable $W$ with distribution $P_W$ such that 1) the distribution of the quantization output $X$ that can take $k$ values follows any specified distribution $P_X$ over $\{1,\ldots,k\}$, and 2) the minimum mean squared error (MMSE) of estimating $W$ from $X$ is minimized.
Hugging Face Trending Papers
2d ago
Interpretable intrinsic dimension estimation through componentwise calibration of distance and angle
DANCo (Dimensionality from Angle and Norm Concentration) jointly calibrates nearest-neighbor distance and angular statistics and consistently reaches state-of-the-art accuracy on clean intrinsic-dimen...
arXiv Machine Learning
Aug 27
Transforms for LLM Quantization: The Great Inversion and Format Co-Design
The paper surveys the use of linear, function‑preserving transforms in 4‑bit large‑language‑model (LLM) quantization, formalizing the underlying principle as the "Great Inversion"—the trade‑off between energy concentration favored by allocation‑flexible coding and within‑group flattening favored by grouped shared‑scale quantization. It reviews 200 works, classifies 43 transform methods by structure, data‑awareness, construction approach, and runtime cost, and examines how they interact with GPTQ rounding. The study also explores how different number formats (FP4, MXFP4, NVFP4) influence the optimal transform choice and outlines open research problems. "whyItMatters":"The survey clarifies the conflicting objectives in transform‑based LLM quantization and provides a practical guide for selecting transforms based on deployment regime, thereby informing future research and deployment strategies."
arXiv Machine Learning
1d ago
Interpretable intrinsic dimension estimation through componentwise calibration of distance and angle
arXiv:2609.37114v1 Announce Type: new Abstract: DANCo (Dimensionality from Angle and Norm Concentration) jointly calibrates nearest-neighbor distance and angular statistics and consistently reaches s...
arXiv Machine Learning
Jun 18
Fisher Width: A Geometric Measure of Complexity on Statistical Manifolds
arXiv:2606. 18306v1 Announce Type: new Abstract: Gaussian width is a central geometric complexity measure in high-dimensional probability, compressed sensing, convex optimization, and learning theory.
arXiv Machine Learning
Sep 10
A Nuclear-Norm Lower Bound for Dithered Scalar Quantization of Matrix Products
arXiv:2609.05641v1 Announce Type: cross Abstract: We consider the problem of minimizing error in quantized matrix multiplication $C=AB$. Scalar quantization of the factors introduces rounding errors...
arXiv AI
Aug 13
Causal inference for group-contaminated structured outcomes: observable quotients, lossless reduction and exact randomization inference
arXiv:2608. 11954v1 Announce Type: cross Abstract: Structured potential outcomes such as microscopy images may be recorded after an unknown, unit-specific transformation.
arXiv Machine Learning
2d ago
Cosine Similarity Is Not Evidence: Measuring the Noise Floor of Interpretability Transfer Under Quantization
The paper argues that reporting scale‑invariant statistics such as cosine similarity without their noise floor is misleading when evaluating interpretability transfer from full‑precision to quantized neural networks. It derives a closed‑form expression for the expected cosine similarity based on a dimensionless parameter κ = n ho^2/d, measures the class separation ρ on real activations, and shows that a reported cosine of 0.996 between full‑precision and INT4 models cannot be interpreted as preservation without knowing the sample size n. The authors demonstrate that at INT4 the direction of the interpretability artifact rotates beyond the estimator’s own noise, while at INT8 no significant movement is detected, and they highlight that scale‑invariant metrics cannot distinguish between translation and attenuation of a transferred decision variable.