The paper introduces a quantization technique for tabular foundation models that focuses on converting queries, keys, and values to FP8 and employing explicit FP8 matrix multiplication to accelerate attention calculations. It emphasizes aligning quantization errors between training and test rows to avoid accuracy loss, and demonstrates up to 1.7× speedup over 16‑bit kernels with no significant accuracy degradation on TabPFN‑v3 and TabICLv2 across TabArena and BeyondArena.
HyQuant introduces a hybrid-precision quantization framework for large language model (LLM) attention modules. It quantizes most attention states to low-bit formats while preserving a small set of vertical‑line tokens and local‑window states in full precision, guided by lightweight attention‑pattern signals. This design achieves near‑lossless accuracy across tasks while improving memory and hardware efficiency.
The paper introduces a reproducible benchmark for evaluating attention mechanisms in tabular foundation models, focusing on the distinct row and column attention patterns that differ from language model attention. It compares several backends—Torch SDPA, FlashAttention variants, vLLM, and SageAttention—across realistic tabular shapes on A100, H100, and B200 GPUs, revealing that optimal backend choice varies by attention type, hardware, and model specifics. The study finds FlashAttention generally performs best, but CuDNN can outperform it for column attention on longer sequences, while SageAttention excels for large row sequences beyond 16k rows.
arXiv:2606. 31519v1 Announce Type: new Abstract: Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased.
arXiv:2606. 04620v1 Announce Type: cross Abstract: LLMs have become the state-of-the-art algorithms for solving NLP tasks.
arXiv:2609.38166v1 Announce Type: cross Abstract: Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Atte...
arXiv:2608. 02691v1 Announce Type: cross Abstract: The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important.
arXiv:2606. 07345v1 Announce Type: new Abstract: Tabular foundation models, exemplified by TabPFN, perform prediction via in-context learning, inferring test labels directly from labeled training examples.
The paper presents a comprehensive evaluation of post‑training quantization (PTQ) techniques for large language models (LLMs), addressing gaps in prior work by conducting fair, large‑scale experiments on text‑only LLMs. It decouples quantization into pre‑quantization transformation and error mitigation, systematically studies granularity, symmetry, and data formats (MXFP4, NVFP4), and finds that rotation‑based transformations and low‑rank compensation with GPTQ can improve performance, while finer granularity and asymmetric quantization benefit activations more than weights. The study highlights the importance of understanding component interactions and the impact of scaling factor formats on low‑bit quantization performance.
RAQ-MLA introduces a role-aware quantization scheme for Multi-Head Latent Attention (MLA) caches used in large language models. It analyzes the distinct functions of the cached latent and positional key tensors, applying tailored invertible transforms to each based on their impact on attention output. The method achieves near-lossless performance while reducing the cache size to about 2.5 bits per element, effectively increasing theoretical KV-cache capacity by 6.5×.
arXiv:2609.00224v1 Announce Type: cross Abstract: Weight-only post-training quantization (PTQ) can alleviate the computational burden of serving large language models (LLMs) at scale. However, existi...
arXiv:2609.36760v1 Announce Type: new Abstract: Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memor...
arXiv:2610.27699v1 Announce Type: new Abstract: Low-bit quantization reduces the memory footprint of large language models, but preserving accuracy with efficient training remains challenging. Scalar...