← Back to all news
arXiv Computer Vision By Suho Yoo, Hyunjong Ok, Jongmin Choi, Jihoo Jung, Joon Son Chung

OmniKVQuant: KV Cache Quantization for Omni-LLMs

Read the original on arXiv Computer Vision →

OmniKVQuant introduces a training‑free framework for quantizing the key‑value (KV) cache of omni‑modal large language models (Omni‑LLMs) that process audio, video, and text simultaneously. The method addresses two identified problems—temporal key drift and heterogeneous value geometry—by setting key quantization ranges over short input windows and rotating values separately for each modality. Applied to Qwen2.5‑Omni and Qwen3‑Omni, OmniKVQuant achieves 2‑bit KV caches while largely preserving performance across seven audio‑visual benchmarks, and includes a fused Triton decode kernel that eliminates the need for a dense FP16 cache.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv AI
1d ago

HyQuant: Hybrid-Precision Quantization for LLM Attention

HyQuant introduces a hybrid-precision quantization framework for large language model (LLM) attention modules. It quantizes most attention states to low-bit formats while preserving a small set of vertical‑line tokens and local‑window states in full precision, guided by lightweight attention‑pattern signals. This design achieves near‑lossless accuracy across tasks while improving memory and hardware efficiency.

By Jiatong Ding, Bingxin Xing, Yu Zhang, Dian Ding, Xiaodong Yi, Xianbin Ouyang, Feihu Zhou, Kun Zhang, Zhenyu Guo, Hao Pan, Guangtao Xue, Yiming Zhang
llmsefficiency
More like this →