The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
FIGFLEX is a mixed‑precision accelerator co‑designed with a low‑bit LLM quantizer that performs floating‑point‑integer (FP‑INT) GEMM. It introduces a shift‑add, radix‑4 reformulation of FP‑INT multiplication that fuses multiply‑accumulate into a precision‑flexible partial‑product reduction, and a novel INT3 optimization that reclaims idle datapath leaves. The accompanying mixed‑precision quantizer assigns channel‑group bit‑widths based on GPTQ error, using power‑of‑two column scaling to keep activations out of the FP16 subnormal range, enabling the quantized model to run on FIGFLEX with the same accuracy as on a GPU while achieving higher throughput per area on 3‑bit mixed‑precision workloads.
ShatterQuant is a hardware-software co-designed framework that enables mixed-precision quantization within individual tensors by assigning different bit-widths to blocks of a weight projection. It couples precision granularity with processing element configuration, allowing each precision to determine an effective block height. The framework includes a hardware-aware post-training method based on block-level standard deviation and weight sensitivity, a ShatterQuant Transformer Accelerator supporting 1/2/4/8-bit weight precision, precision-dependent PE configuration, block rescaling, and integrated softmax and piecewise-linear nonlinearities, and an evaluation showing 1.5 TOPS, 760 GOPS/$mm^2$ area efficiency, and 2.8 TOPS/W energy efficiency on a TSMC 16nm PDK implementation.
arXiv:2611.08363v1 Announce Type: new Abstract: Quantization reduces the memory and bandwidth demands of large language model (LLM) inference but can degrade accuracy; offloading higher-precision dat...
arXiv:2607. 14618v1 Announce Type: new Abstract: CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs.
arXiv:2608. 08081v1 Announce Type: cross Abstract: Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand.
arXiv:2602. 01027v2 Announce Type: replace Abstract: Mixed-precision quantization is a promising approach for compressing large language models under tight memory budgets.