arXiv AI
RECAP: Calibrating Low-Bit Reasoning Models on the History They Read
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Aug 7
QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
arXiv:2608. 05326v1 Announce Type: new Abstract: Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache.
arXiv AI
Jun 2
Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery
arXiv:2606. 02011v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) rely on long reasoning traces, making inference expensive.
Hugging Face Trending Papers
Jun 1
Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery
Large Reasoning Models (LRMs) rely on long reasoning traces, making inference expensive. While low-bit quantization reduces per-token decoding cost, we show that aggressive 2-bit inference can fail to deliver end-to-end speedup because instability in the generation process inflates total token count.
arXiv Machine Learning
Oct 22
Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
The paper introduces OnQ, a low‑bit recovery pipeline for quantized large language models that uses on‑policy distillation and candidate‑wise Dynamic KL to address trajectory divergence and distributional misalignment. OnQ starts with block‑wise calibration, then lets the quantized student generate responses and receive full‑precision teacher supervision on its own prefixes, dynamically weighting forward and reverse KL losses per candidate. Experiments on Qwen3 models show that OnQ outperforms baselines on nine benchmarks, especially long‑horizon reasoning, and produces shorter, less truncated responses.
arXiv Machine Learning
Sep 7
BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
BeaconKV is a training‑free key‑value cache compression technique for Large Reasoning Models that uses beacon queries—compact representatives of query clusters—to predict which KV pairs will be revisited during long‑horizon reasoning. By focusing on Thought Revisiting Tokens that re‑attend distant context, BeaconKV reduces memory usage up to 5.8× and improves throughput by over 4.3× while largely preserving cache accuracy across multiple open‑source LRMs and reasoning benchmarks.
arXiv Machine Learning
Jun 26
What Survives When You Compress a Recursive Reasoner for the Edge?
arXiv:2606. 26488v1 Announce Type: new Abstract: Recursive reasoning models can solve complex structured tasks with only a few million parameters by repeatedly updating a latent state.