arXiv Computation and Language
WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Aug 11
VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference
arXiv:2608. 08569v1 Announce Type: new Abstract: Recent advancements in Speech Large Language Models have demonstrated remarkable capabilities in understanding complex audio tasks.
arXiv AI
Aug 26
Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention
Minima-KV introduces a retention‑preserving hierarchy for mixed‑format paged attention that keeps recent and protected anchor pages in FP8 while older pages are compressed into packed TQ3, allowing every live‑request page to remain addressable. The approach uses format‑specific kernels and a globally normalized online‑softmax merge to compute partial attention states, enabling direct heterogeneous decoding without a dense shadow cache. Experiments on Qwen3.6‑27B on a 96‑GB NVIDIA RTX PRO 6000 Blackwell GPU show 3.50× compression over BF16 and 1.75× over FP8, with minimal impact on performance across long‑context benchmarks.
arXiv AI
6d ago
Acoustic-to-Text KV Compression for Full-Duplex Speech Models
The paper introduces acoustic-to-text KV compression for full‑duplex speech models, converting acoustic key‑value states into compact textual memory during listening‑time slack. When the KV cache exceeds a target budget, older acoustic states are evicted while transcripts and recent acoustic context are retained. Experiments on ten‑minute LongSpeech sessions show a 64.6% reduction in peak streaming KV‑cache size and improved transcription, temporal question answering, and summarization, with comparable pause‑handling, turn‑taking, and interruption performance in Full‑Duplex‑Bench.
arXiv Machine Learning
Jul 27
HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
arXiv:2607. 22389v1 Announce Type: cross Abstract: With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck.
arXiv Computer Vision
Sep 11
OmniKVQuant: KV Cache Quantization for Omni-LLMs
OmniKVQuant introduces a training‑free framework for quantizing the key‑value (KV) cache of omni‑modal large language models (Omni‑LLMs) that process audio, video, and text simultaneously. The method addresses two identified problems—temporal key drift and heterogeneous value geometry—by setting key quantization ranges over short input windows and rotating values separately for each modality. Applied to Qwen2.5‑Omni and Qwen3‑Omni, OmniKVQuant achieves 2‑bit KV caches while largely preserving performance across seven audio‑visual benchmarks, and includes a fused Triton decode kernel that eliminates the need for a dense FP16 cache.
arXiv Machine Learning
Sep 21
TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching
arXiv:2609.21172v1 Announce Type: new Abstract: Large language models (LLMs) are moving onto mobile devices for increasingly diverse workloads over text, images, video, and audio. These applications...