Hugging Face Trending Papers
Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv AI
Sep 17
The Inference Engineering Pareto Atlas: Which Optimizations Dominate the Cost, Quality, and Latency Frontier?
The paper presents a Pareto atlas of LLM inference optimizations, mapping cost, quality, and latency trade‑offs for Qwen2.5‑7B‑Instruct on L4, A100, and H100 GPUs. Using 54 measured configurations and a calibrated simulator, it identifies 18 of 36 setups on the Pareto frontier, showing that combined methods outperform single ones. Quality tests reveal that AWQ 4bit and FP8 weights offer significant latency reductions while largely preserving accuracy, but naive FP8 KV caching fails to answer any questions correctly.
Hugging Face Trending Papers
Oct 5
StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs
StagQ is a multi‑precision weight format for large language models that uses a 2‑bit group‑wise affine base followed by optional 1‑bit refinement planes. Each supported precision can be read as a prefix of the main stream, decoded via a shared affine map without per‑weight lookups, and a sparse side record stores the few weights that the grid handles poorly. Experiments show that StagQ outperforms baseline multi‑precision schemes on Llama‑3.1‑8B, Phi‑4, and OLMo‑2‑7B across various bit‑widths, and its GPU kernel is faster than baseline kernels for most shape‑precision combinations.
arXiv Machine Learning
Aug 26
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning
arXiv:2608.23816v1 Announce Type: new Abstract: Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We pres...
arXiv Machine Learning
1d ago
StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference
arXiv:2611.08363v1 Announce Type: new Abstract: Quantization reduces the memory and bandwidth demands of large language model (LLM) inference but can degrade accuracy; offloading higher-precision dat...
arXiv Machine Learning
Sep 1
SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
arXiv:2608.28911v1 Announce Type: new Abstract: The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length....
arXiv AI
Oct 26
Training an MoE LLM in MXFP4: How We Closed the Gap to BF16
The paper introduces ALTO, an open‑source training recipe that enables a 20‑billion‑parameter mixture‑of‑experts LLM (GPT‑OSS‑20B) to train on the MLPerf Small MoE benchmark using the MXFP4 sub‑8‑bit format. By combining hybrid 1D/2D block quantization, randomized Hadamard transforms, stochastic rounding, and a simplified weight de‑oscillation scheme, the authors achieve a validation loss of 3.3350 after 16,128 steps—only 0.007 above the BF16 baseline—while reducing the FP4 convergence overhead from +20.0% to +5.0% relative to BF16. The study also reports negative results for differential gradient estimation, outlier clipping, and macro‑block scaling, despite operator‑level SNR improvements, and notes that the experiments use simulated MXFP4 kernels on AMD MI300 hardware without wall‑clock optimization.