arXiv Machine Learning August 11, 2026 By Anthony. Lui, Mohamed. Elsaied, N. P. Savani

RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

Read the original on arXiv Machine Learning →

arXiv:2608. 08081v1 Announce Type: cross Abstract: Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 2

BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization

arXiv:2606. 00079v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) large language models reduce per-token computation through sparse expert activation, but their deployment remains memory-intensive because all expert weights must be kept resident in memory.

By Jiayu Zhao, Zihan Teng, Minhao Fan, Tianrui Ma, Wentao Ren, Song Chen, Weichen Liu

llms efficiency

arXiv AI

Jun 19

UltraQuant: 4-bit KV Caching for Context-Heavy Agents

arXiv:2606. 20474v1 Announce Type: cross Abstract: Context-heavy agents place unusual pressure on the key-value (KV) cache: long prefixes are reused across many short turns, while concurrency determines whether the serving system can keep GPUs utilized.

By Inesh Chakrabarti (Advanced Micro Devices, University of California, Los Angeles), David Limpus (Advanced Micro Devices, Purdue University), Aditi Ghai Rana (Advanced Micro Devices), Bowen Bao (Advanced Micro Devices), Spandan Tiwari (Advanced Micro Devices), Thiago Crepaldi (Advanced Micro Devices), Ashish Sirasao (Advanced Micro Devices)

agents efficiency

arXiv Machine Learning

Jun 24

RoPE-Aware Bit Allocation for KV-Cache Quantization

arXiv:2606. 24033v1 Announce Type: new Abstract: Existing low-bit KV-cache quantizers often treat each cached key as a flat vector.

By Fengfeng Liang, Yuechen Zhang, Jiaya Jia

llms efficiency

arXiv Machine Learning

Jun 25

SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

arXiv:2605. 01708v3 Announce Type: replace-cross Abstract: Contemporary systems serving large language models (LLMs) have adopted prefill-decode disaggregation to load-balance between the compute-bound prefill phase and the memory-bound decode phase.

By Yipin Guo, Siddharth Joshi

llms agents efficiency

arXiv AI

Jun 4

LiftQuant: Continuous Bit-Width LLM via Dimensional Lifting and Projection

arXiv:2606. 04050v1 Announce Type: cross Abstract: Existing quantization methods are fundamentally limited by rigid, integer-based bit-widths (e.

By Liulu He, XuanAng Liu, Juntao Liu, Taolue Feng, Ting Lu, Chunsheng Gan, Zhiyv Peng, Yuan Du, Huanrui Yang, Yijiang Liu, Li Du

llms efficiency benchmarks

arXiv Machine Learning

Jun 29

RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory

arXiv:2605. 06675v2 Announce Type: replace Abstract: Large language models cache all previously computed key-value (KV) pairs during generation, and this KV cache grows linearly with sequence length, making it a primary memory bottleneck for serving.

By Fei Zuo, Zikang Zhou, Hao Cong, Xiaoyan Xi, Ho Fai Leung

llms efficiency

Related stories

BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization

UltraQuant: 4-bit KV Caching for Context-Heavy Agents

RoPE-Aware Bit Allocation for KV-Cache Quantization

SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

LiftQuant: Continuous Bit-Width LLM via Dimensional Lifting and Projection

RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory