OmniKVQuant: KV Cache Quantization for Omni-LLMs

OmniKVQuant introduces a training‑free framework for quantizing the key‑value (KV) cache of omni‑modal large language models (Omni‑LLMs) that process audio, video, and text simultaneously. The method addresses two identified problems—temporal key drift and heterogeneous value geometry—by setting key quantization ranges over short input windows and rotating values separately for each modality. Applied to Qwen2.5‑Omni and Qwen3‑Omni, OmniKVQuant achieves 2‑bit KV caches while largely preserving performance across seven audio‑visual benchmarks, and includes a fused Triton decode kernel that eliminates the need for a dense FP16 cache.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

• llms
• efficiency
• multimodal
• benchmarks