arXiv AI
TriAttention: Unified Sparse Mixed-Precision Attention for Long-Context LLM Inference
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 19
TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration
TileMix is a tile‑centric mixed‑precision attention kernel that routes score‑tile groups within fused dense attention to either FP16 or INT8 computation, using compact bitmasks to decide precision per tile. By partitioning the attention matrix into hardware‑aligned tiles and updating a shared online‑softmax state, TileMix preserves dense token connectivity without requiring training and supports grouped‑query attention, variable‑length batches, and INT8 key/value caches. Benchmarks on LLaMA, Qwen, and Vicuna show that TileMix restores long‑context quality lost with uniform INT8 and improves prefill throughput over FP16, offering a controllable accuracy‑efficiency trade‑off across model families.
arXiv AI
Oct 1
ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
arXiv:2602.08005v2 Announce Type: replace-cross Abstract: Efficient long-context inference faces two coupled bottlenecks: KV-cache memory grows linearly with context length, while attention computati...
arXiv Machine Learning
Jun 9
SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inference
arXiv:2605. 18856v3 Announce Type: replace Abstract: Long-context inference is increasingly constrained by the KV cache: resident memory grows with context length, and decoding becomes limited by repeated High Bandwidth Memory (HBM) streaming rather than arithmetic.
arXiv Machine Learning
Jun 3
Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
arXiv:2606. 02964v1 Announce Type: cross Abstract: Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation.
arXiv Machine Learning
Oct 9
Read What Matters: Query-Adaptive Quantization for KV Caches
The paper introduces ReadKV, a query‑adaptive quantization method for KV caches that stores keys and values in progressive codes, allowing each decoding query to allocate different precision prefixes for keys and values. By optimizing a calibrated distortion objective under fixed budgets, ReadKV achieves near‑optimal allocation and demonstrates that query‑dependent access can outperform any query‑independent reader at the same read budget. Experiments on six base models show that reading only four bits on average from an eight‑bit cache increases perplexity by at most 0.66% while reducing logical reads and retained capacity compared to a 16‑bit cache, and it achieves lower latency than existing codecs on an NVIDIA A10G.
arXiv Machine Learning
Sep 30
QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching
arXiv:2609.36760v1 Announce Type: new Abstract: Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memor...