arXiv AI
Fewer Bits, More Tries: Recovering Capability Under Memory Constraints
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 30
Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inference
arXiv:2609.36654v1 Announce Type: new Abstract: Large language models make weight storage and memory traffic major inference costs, motivating low-precision formats that represent each weight with on...
arXiv AI
Jul 21
KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?
arXiv:2607. 16241v1 Announce Type: cross Abstract: Recent large language models (LLMs) can generate custom CUDA kernels that appear to outperform PyTorch on benchmarks such as KernelBench.
arXiv Computation and Language
Sep 1
Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
The paper presents a budget‑aware compression pipeline for deploying 70B‑parameter language models on a single NVIDIA GPU. It examines how pruning, quantization, and KV‑cache compression interact, showing that layer‑wise pruning improves weight quantization robustness and that KV‑cache sparsification complements INT8 KV quantization without harming decoding speed. Using these insights, the authors compressed a 70B model to ~33 GB, achieving ~57 tokens/s on 10k‑token prompts on an A40 while maintaining accuracy within 5% on standard benchmarks.
arXiv AI
Jul 17
Smarter and Cheaper at Once: Byte-Exact KV-Cache Grafting Turns a Frozen Small Model into a Verified-Knowledge Flywheel
arXiv:2607. 14431v1 Announce Type: cross Abstract: We report a way to make a frozen small language model both more capable and dramatically cheaper at once, without changing any weights.
arXiv AI
Oct 9
Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute
The paper presents a memory layer called galahad-kv that stores the key-value state of a large language model’s context blocks (≈16,000 tokens each) on encrypted local NVMe disk, allowing byte‑exact retrieval without recomputation. On a single NVIDIA H100, the authors tested the system with Gemma 4 12B and 31B models over 50 million tokens of real public text, achieving 100 % successful reloads, 2.8×–4.3× faster loading than recomputation, and 8.8×–12.3× lower GPU energy usage while keeping GPU memory flat. The models answered long‑range factual queries with high accuracy (82 % for 12B, 98 % for 31B) and did not hallucinate, demonstrating that stored state can extend effective context without expanding the attention window.
arXiv Computation and Language
Sep 23
Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs
Flash-dLLM is a training‑free inference acceleration framework that improves the speed and memory efficiency of Diffusion Large Language Models (dLLMs). It tackles GPU memory I/O bottlenecks by introducing an I/O‑aware fused KV‑cache kernel and then employs a draft‑and‑verify decoding strategy that uses the dLLM itself as both drafter and verifier. Experiments on mathematical reasoning and code‑generation tasks show Flash‑dLLM outperforms existing acceleration methods, achieving up to 11.0× speedups over the Elastic‑Cache baseline.