← Back to all news
arXiv AI By Muhammad Azlan Qazi, Alexandros Iosifidis, Qi Zhang

TRACE: Transformer Residual Activation Codebook Encoding for Communication-efficient Distributed Edge Inference

Read the original on arXiv AI →

The paper introduces TRACE, a framework that embeds a Vector Quantizer at the post‑attention residual of each Transformer layer, enabling exchanged activations to be encoded as compact codebook indices. During inference, receivers recover key/value representations via table lookup, eliminating downstream computation and allowing communication–computation overlap. A hierarchical codebook further permits runtime bitrate adaptation, and experiments on ViT‑Base and ViViT‑B demonstrate up to 1.89% accuracy gain, 1.27× speedup, and a 22,624× reduction in communication overhead compared to uncompressed inference.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Sep 25

FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates

FlashLoop is a training‑free inference framework for Looped Transformers that reduces cross‑loop redundancy by employing token‑sparse updates, sparse attention, and KV‑residual quantization. It exploits observations that, as loops progress, state changes concentrate on a small token subset, attention differences are dominated by a sparse key subset, and KV residuals become amenable to low‑bit quantization. The method achieves lossless accuracy with up to 1.64× speedup and 6× KV‑cache memory reduction across several Looped Transformer models.

By Wanqi Yang, Shiwei Liu
llmsefficiency
More like this →
arXiv Machine Learning
Oct 8

ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals

ResidualQuant introduces a 2‑bit residual quantization scheme for KV caches in looped Transformers, leveraging the similarity of KV states across loops. By using the final‑loop KV states as a reference and applying least‑square scaling, rotations, and loop‑wise mixed precision, the method achieves accurate quantization down to INT2. Experiments on various looped Transformer models show that ResidualQuant improves the accuracy‑memory tradeoff, retaining near BF16 accuracy while reducing KV storage by 80.7% and boosting decode throughput and batch size on an RTX 5090.

By Heejun Kim, Junyoung Lee, SangLyul Cho, Dongsu Han, Insu Han, Sehoon Kim
llmsefficiencybenchmarks
More like this →
arXiv Machine Learning
Sep 22

PRQuant: Permutation Residual Quantization for Low-Overhead Inference

PRQuant introduces a training‑free, low‑overhead method for low‑bit quantization of linear layers by permuting input channels that cause the largest quantization error into contiguous tail blocks and precomputing residual weight sub‑tensors. The approach eliminates the need for online gathering during inference, converting scattered residual compensation into a regular tail‑augmented GEMM and thereby reducing latency. Experiments show that PRQuant lowers down‑projection reconstruction error and outperforms standard MXFP4 and other post‑training quantization baselines on five downstream benchmarks, improving accuracy by up to 1.24 points on Qwen3‑4B‑Instruct‑2507.

By Peiran Wang, Anqi Wang, Jiaying Zhao, Huiwen Yang, Zhenyu Ming, Rongqian Wang, Yiwu Yao, Kun Tian, Xin Yao, Gong Zhang, Fan Yang, Zhongyi Huang
efficiencybenchmarks
More like this →