arXiv AI
Jun 2
Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery
arXiv:2606. 02011v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) rely on long reasoning traces, making inference expensive.
ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Training
arXiv:2606. 15682v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) achieve strong problem-solving through long chain-of-thought, but their deployment is constrained by the high cost of full-precision inference and growing KV cache footprints.
Related stories
arXiv AI
Jun 12
ReSET: Accurate Latency-Critical NVFP4 Reasoning via Step-Aware Temperature Scaling
arXiv:2606. 13233v1 Announce Type: cross Abstract: Large reasoning models (LRMs) improve complex problem-solving by generating long intermediate reasoning traces, but this substantially increases inference costs.
Hugging Face Trending Papers
Jun 1
Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery
Large Reasoning Models (LRMs) rely on long reasoning traces, making inference expensive. While low-bit quantization reduces per-token decoding cost, we show that aggressive 2-bit inference can fail to deliver end-to-end speedup because instability in the generation process inflates total token count.
arXiv Machine Learning
Jun 26
What Survives When You Compress a Recursive Reasoner for the Edge?
arXiv:2606. 26488v1 Announce Type: new Abstract: Recursive reasoning models can solve complex structured tasks with only a few million parameters by repeatedly updating a latent state.
arXiv Machine Learning
Sep 23
Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning
arXiv:2609.26708v1 Announce Type: new Abstract: Quantization-aware distillation (QAD) restores much of the short-form question-answering performance lost to sub-3-bit quantization, yet leaves mathema...
arXiv AI
Jul 29
Stable FP4 Training via Transposition-Invariant Block Quantization
arXiv:2607. 24953v1 Announce Type: cross Abstract: Reducing training precision is a key lever for improving the e ciency of large language model (LLM) training, but pushing beyond FP8 to 4-bit oating point (FP4) remains challenging due to instability during optimization.
arXiv Machine Learning
Jun 25
Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models
arXiv:2606. 25519v1 Announce Type: cross Abstract: Quantization is widely used to reduce the inference cost of large language models, but its effect on reasoning models is not fully captured by final-answer accuracy or per-token latency.
arXiv AI
Sep 10
Reasoning-Aware Compression: Identifying and Protecting Vulnerable Reasoning Circuits for Energy-Efficient LLM Deployment
The paper introduces a reasoning‑aware compression framework for Large Reasoning Models (LRMs) that identifies vulnerable reasoning circuits and selectively restores them to FP16 to avoid performance loss. By benchmarking INT4 quantization across five reasoning tasks (GSM8K, FOLIO, MATH‑500, ProofWriter, MuSiQue) and measuring GPU energy, the authors find that uniform quantization can actually increase energy consumption and that vulnerability varies by task and architecture. Selective compression yields Pareto‑optimal trade‑offs, achieving significant energy savings while improving or matching accuracy on held‑out data.
arXiv Machine Learning
Sep 7
BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
BeaconKV is a training‑free key‑value cache compression technique for Large Reasoning Models that uses beacon queries—compact representatives of query clusters—to predict which KV pairs will be revisited during long‑horizon reasoning. By focusing on Thought Revisiting Tokens that re‑attend distant context, BeaconKV reduces memory usage up to 5.8× and improves throughput by over 4.3× while largely preserving cache accuracy across multiple open‑source LRMs and reasoning benchmarks.
arXiv Machine Learning
2d ago
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
arXiv:2609.39223v2 Announce Type: new Abstract: Large language model (LLM) inference is increasingly moving toward lower precision to realize the throughput of hardware accelerators, but aggressive p...
arXiv Machine Learning
Jun 3
Value-Aware Stochastic KV Cache Eviction for Reasoning Models
arXiv:2606. 03928v1 Announce Type: new Abstract: Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck.
arXiv Machine Learning
Aug 31
DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
The paper introduces DAMP, a decay‑aware mixed‑precision quantization scheme for recurrent‑state representations in GDN and KDA language models. By identifying high‑risk channels through quantization‑error energy and decay persistence, DAMP stores these channels at higher precision while compressing the rest to INT8, achieving a 9.9‑bit average precision. Experiments on Qwen3.6‑35B and Kimi‑Linear‑48B show a 69.1% reduction in recurrent‑state storage, up to 2.01× faster state‑update kernels, and up to 10.9% lower full‑model TPOT while preserving accuracy close to the FP32 baseline.