CoS‑VQ is a new vector‑quantization technique for large language models that keeps codewords low‑dimensional while jointly optimizing their hard indices and a shared scale across positions. By collecting candidates at multiple probe scales and using a tangent separable upper bound, it reduces candidate comparisons to a small set of representative intervals, enabling efficient joint index–scale updates without exhaustive enumeration. Experiments on Qwen3.8‑27B show that CoS‑VQ outperforms existing methods such as QTIP and LLVQ on language‑modeling, coding, and reasoning benchmarks, and ablations demonstrate that joint updates yield larger gains than scale‑only refinement.
arXiv:2607. 01065v1 Announce Type: new Abstract: The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory.
JARQ is a plug‑in refinement method for group‑wise post‑training quantizers that alternates a joint least‑squares fit of all group scales with bounded Babai proposals to move many codes together on the current grid. It solves a bilinear box‑constrained mixed‑integer least‑squares problem without backpropagation, preserves the host model’s bit width, groups, zero points, and inference cost, and improves accuracy across Llama‑2, Llama‑3, and Qwen models when paired with RTN, GPTQ, OmniQuant, and AWQ hosts. In practice, JARQ reduces perplexity in 90 of 96 comparisons, cuts three‑bit RTN perplexity by up to 36%, raises mean multiple‑choice accuracy in 23 of 24 configurations, and enhances QEP, QuaRot, and OJBKQ outputs, all within under a minute per 7B block.
arXiv:2606. 03458v1 Announce Type: new Abstract: Test-time scaling is a powerful approach to obtain better reasoning in large language models, but it becomes memory-bottlenecked during long-horizon decoding, as the KV-cache grows.
arXiv:2610.05265v2 Announce Type: replace Abstract: Looped language models provide a parameter-efficient way to scale iterative test-time computation by repeatedly executing a shared recurrent core....
arXiv:2505.10202v2 Announce Type: replace Abstract: Large Language Models (LLMs) have achieved remarkable success but face significant computational and memory challenges, particularly due to their e...
arXiv:2608.30384v1 Announce Type: new Abstract: By introducing RSLM (Rotated Scaled Lloyd-Max), a family of training-free vector quantization codecs compressing embeddings to 1--4 bits per dimension,...
arXiv:2605. 08692v2 Announce Type: replace Abstract: Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference.
arXiv:2608.30908v1 Announce Type: new Abstract: Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form. This setting is practi...
arXiv:2607. 15498v1 Announce Type: cross Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference.
arXiv:2610.27699v1 Announce Type: new Abstract: Low-bit quantization reduces the memory footprint of large language models, but preserving accuracy with efficient training remains challenging. Scalar...
MultiAlloc is a profiling‑guided framework that allocates limited quantization resources between layer‑wise precision enhancement and token‑wise preservation for KV cache quantization in large language models. By using lightweight offline profiling to assess layer sensitivity and token importance, it automatically derives an effective precision allocation strategy that satisfies a fixed memory budget without online search. Experiments on reasoning and long‑context benchmarks show that MultiAlloc consistently outperforms existing layer‑wise, token‑wise, and multi‑granularity quantization methods across models such as Mistral, Llama, DeepSeek‑Qwen, and QwQ.
arXiv:2610.03027v1 Announce Type: cross Abstract: The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwid...