← Back to all news
arXiv AI By Hanzhi Zhang, Qiao Zhang, Qinglei Cao, Heng Fan, Yan Huang, Kewei Sha, Yunhe Feng

TileQuant: Tile as the Precision-Allocation Unit for LLM Mixed-Precision Post-Training Quantization

Read the original on arXiv AI →

TileQuant is a training‑free post‑training quantization framework that assigns precision to 2‑D weight tiles aligned with hardware. It uses offline calibration to score tiles based on activation statistics and loss sensitivity, keeping the highest‑scoring tiles at 16‑bit while quantizing the rest to 8‑bit. The resulting precision map enables efficient integer and 16‑bit kernels that reduce memory traffic and achieve 2.66× the original model’s generation throughput while preserving or surpassing quality across multiple language‑modeling tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Aug 19

TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration

TileMix is a tile‑centric mixed‑precision attention kernel that routes score‑tile groups within fused dense attention to either FP16 or INT8 computation, using compact bitmasks to decide precision per tile. By partitioning the attention matrix into hardware‑aligned tiles and updating a shared online‑softmax state, TileMix preserves dense token connectivity without requiring training and supports grouped‑query attention, variable‑length batches, and INT8 key/value caches. Benchmarks on LLaMA, Qwen, and Vicuna show that TileMix restores long‑context quality lost with uniform INT8 and improves prefill throughput over FP16, offering a controllable accuracy‑efficiency trade‑off across model families.

By Hanzhi Zhang, Qiao Zhang, Qinglei Cao, Heng Fan, Yan Huang, Kewei Sha, Yunhe Feng
llmsefficiencybenchmarks
More like this →
arXiv Computation and Language
2d ago

MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

MoBiQuant introduces a Mixture-of-Bits quantization framework that allows large language models to be inferred at varying precisions during runtime. It addresses the poor generalization of existing post‑training quantization methods by mitigating precision‑dependent outlier migration through a recursive residual quantization scheme and a token‑aware router that selects the optimal precision per token. Experiments demonstrate that MoBiQuant matches or exceeds single‑precision PTQ performance while providing significant memory savings and up to 1.34× throughput gains over current any‑precision approaches.

By Dongwei Wang, Jinhee Kim, Seokho Han, Denis Gudovskiy, Yohei Nakata, Tomoyuki Okuno, KhayTze Peong, Kang Eun Jeon, Jong Hwan Ko, Yiran Chen, Huanrui Yang
llmsefficiencybenchmarks
More like this →
arXiv Machine Learning
Jun 2

WINDQuant: Weight-Informed Neural Decision-Making for Global Mixed-Precision LLM Quantization

arXiv:2605. 26660v2 Announce Type: replace Abstract: Quantization is an effective approach to reduce the memory footprint and inference cost of large language models (LLMs), yet maintaining performance in the ultra-low-bit regime remains challenging.

By Phong Nam Huu Nguyen, Khoi M. Le, Cong-Duy T Nguyen, Anh Tuan Luu, Thong Thanh Nguyen, Tho Quan
llmsreinforcement-learningefficiency
More like this →