arXiv AI
Jul 1
Hierarchical Global Attention (HGA)
arXiv:2606. 30709v1 Announce Type: cross Abstract: Hierarchical Global Attention (HGA) is a drop-in replacement for dense causal attention in pretrained long-context transformers.
Long-Context Fine-Tuning with Limited VRAM
arXiv:2607. 15105v1 Announce Type: new Abstract: Parameter-efficient fine-tuning reduces model and optimizer memory, but dense attention still makes long training sequences expensive.
Related stories
arXiv AI
Aug 10
Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
arXiv:2604. 26968v2 Announce Type: replace-cross Abstract: Key-value (KV) cache memory management is the primary bottleneck limiting throughput and cost-efficiency in large-scale GPU inference serving.
arXiv AI
Aug 26
Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention
Minima-KV introduces a retention‑preserving hierarchy for mixed‑format paged attention that keeps recent and protected anchor pages in FP8 while older pages are compressed into packed TQ3, allowing every live‑request page to remain addressable. The approach uses format‑specific kernels and a globally normalized online‑softmax merge to compute partial attention states, enabling direct heterogeneous decoding without a dense shadow cache. Experiments on Qwen3.6‑27B on a 96‑GB NVIDIA RTX PRO 6000 Blackwell GPU show 3.50× compression over BF16 and 1.75× over FP8, with minimal impact on performance across long‑context benchmarks.
arXiv Machine Learning
Jul 7
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
arXiv:2603. 17484v2 Announce Type: replace-cross Abstract: Language models struggle to generalize beyond pretraining context lengths, limiting long-horizon reasoning and retrieval.
arXiv AI
Sep 3
HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models
HeadWiseKV is a training‑free framework that compresses the residual global key–value caches of hybrid long‑context language models by assigning each physical KV head a static, multilevel history window. It formulates cache allocation as a restricted operational rate–distortion problem and uses the SeqCalib algorithm to generate per‑head residency policies that account for interactions across layers. In evaluations on four hybrid models, HeadWiseKV preserves near‑full‑KV quality while reducing peak device memory usage by 8.59% at a 112K context length and extending the largest verified context from 114K to 161K.
arXiv AI
Jun 9
FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
arXiv:2606. 09079v1 Announce Type: cross Abstract: Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving.
arXiv Computation and Language
Sep 1
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
arXiv:2608.30320v1 Announce Type: new Abstract: We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and a...
arXiv AI
Aug 21
Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference
arXiv:2608. 20210v1 Announce Type: cross Abstract: Small language models are usually built like large ones and then squeezed onto a CPU afterwards.
Hugging Face Trending Papers
Sep 2
HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models
Long-context inference retains a growing key--value (KV) cache during decoding, which consumes substantial GPU memory and can reduce generation throughput. This bottleneck remains in hybrid language m...
arXiv Machine Learning
Sep 17
Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs
The paper introduces the Tri‑Metric Router, a deterministic, training‑free policy that chooses among Raw, Neural, and Lexical pipelines for retrieval‑augmented generation on commodity GPUs. It uses three CPU‑side signals—spatial complexity, syntactic density, and type‑token ratio—to balance VRAM headroom and latency, calibrated on LongBench qasper. The method eliminates out‑of‑memory failures and improves alignment and F1 scores compared to always‑on lexical compression without extra VRAM or training costs.
arXiv AI
Jul 16
Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference
arXiv:2607. 13205v1 Announce Type: cross Abstract: Attention-based KV cache eviction (H2O and its descendants) compresses the memory-constrained state of a long-context model by ranking tokens on accumulated attention mass, treated here as signal energy, and keeping the heaviest.
arXiv AI
Jun 19
StreamKL: Fast and Memory-Efficient KL Divergence for Boosting Attention Distillation
arXiv:2606. 20005v1 Announce Type: cross Abstract: Attention distillation, which trains one attention distribution to match another by minimizing their Kullback-Leibler (KL) divergence, is widely used in knowledge distillation, model compression, continual learning, and sparse-attention LLM training.