ResCom introduces a low‑rank correction that leverages activation rounding residuals to compensate for layer‑output errors in low‑bit weight–activation quantization of large language models. The method selects residual directions based on their impact on output reconstruction and computes the correction in closed form, avoiding gradient training or weight updates. Experiments on Llama‑3‑8B with 4‑bit weights and activations show significant gains: perplexity drops from 7.708 to 7.147, MMLU improves by 2.89 percentage points, and five‑shot GSM8K accuracy rises from 23.96 % to 36.16 %.
arXiv:2610.08164v1 Announce Type: new Abstract: Low-rank quantization error compensation (LQEC) recovers the accuracy lost under aggressive weight quantization by attaching a closed-form rank-$r$ ada...
arXiv:2608.23144v1 Announce Type: cross Abstract: Low-bit weight quantization saves storage but leaves errors that degrade language-model quality. We introduce Activation-Weighted Seeded Residual Cod...
DualQuant introduces a dual‑scale, calibration‑free post‑training quantization framework that jointly optimizes weight‑reconstruction MSE and derives per‑channel activation rescalings, enabling effective 4‑bit weight and activation quantization without calibration data. The method employs a non‑differentiable optimization solved via variable splitting and block coordinate descent, supporting a wide range of formats including MX and NVFP4. Experiments on Llama and Qwen models show DualQuant matches or surpasses calibration‑based SmoothQuant and significantly outperforms other calibration‑free baselines, achieving up to 2.61× memory reduction and 2.46× throughput speedup on an RTX 4090.
Low-rank quantization error compensation (LQEC) recovers the accuracy lost under aggressive weight quantization by attaching a closed-form rank-$r$ adapter beside each frozen quantized weight, without...
arXiv:2605. 08692v2 Announce Type: replace Abstract: Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference.
arXiv:2608. 04048v1 Announce Type: cross Abstract: Serving large language models (LLMs) under diverse deployment constraints requires flexible trade-offs between accuracy, memory footprint, and throughput.
arXiv:2606. 13054v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment.
arXiv:2601. 07475v2 Announce Type: replace-cross Abstract: The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference.
Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a promising compression technique, offering significant reductions in model size and inference complexity.
SHIFT-LLM is a training‑free post‑pruning correction framework that inserts a Linear Residual Adapter (LRA) at each depth‑pruned site in large language models. Each LRA preserves the original residual identity while adding a lightweight affine correction calibrated via closed‑form least‑squares regression on a small held‑out set, thereby approximating the hidden state that would have been produced by the removed block. Experiments across multiple model families and benchmarks show that SHIFT‑LLM consistently recovers accuracy lost to depth pruning, achieving gains up to +15.7 points on Llama‑3.1‑8B‑Instruct with only a few hundred calibration samples and no gradient computation.
The paper reports a post‑training ternarization of the 4‑billion‑parameter Qwen model, achieving an effective 1.641‑bit representation for 81.62 % of its weights while keeping activations at 16‑bit precision. Accuracy drops from 64.5 % to 54.7 % across ten capability tests, with uneven degradation (e.g., BoolQ 84.6 % of teacher performance, ARC‑Challenge 43.8 %). After packing the ternary planes, the model size shrinks from 8.29 GiB to 3.96 GiB with negligible change in perplexity, though inference speed is not improved.
arXiv:2606. 07819v1 Announce Type: new Abstract: Recently, the efficiency of Large Language Models (LLMs) deployment has become a critical concern in practical applications.