Hugging Face Trending Papers September 28, 2026

DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory

Read the original on Hugging Face Trending Papers →

DPS (Dual-Mode Precision LLM Serving) is a system that treats model‑weight memory as elastic by using a multi‑precision representation. Under normal load it serves the full‑accuracy model, but when KV‑cache pressure spikes it switches to a lower‑precision variant and reallocates unused weight memory for KV cache blocks. Built on Semi‑Unified Memory and implemented on top of vLLM, DPS boosts sustained throughput by 2.1–3.3× and effective pass@1 by up to +41 pp over static FP16 while maintaining FP16‑class accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

arXiv AI

Jun 24

CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

arXiv:2606. 24506v1 Announce Type: cross Abstract: Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold.

By Zhuoren Ye, Tianyu Wo, Dinghao Xue, Mingming Zhang, Yuchen Teng, Chunming Hu, Renyu Yang

llms benchmarks

arXiv Machine Learning

Jul 20

PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

arXiv:2607. 16184v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy.

By Yuchen Yang, Yifan Zhao, Anisha Dasgupta, Sasa Misailovic

llms efficiency

arXiv Machine Learning

Jun 5

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

arXiv:2606. 06302v1 Announce Type: new Abstract: Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth.

By Hyungmin Kim, Minsoo Kim, Hongseok Kim, Jungwook Choi

llms efficiency

Hugging Face Trending Papers

Jun 4

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth. Non-uniform KV compression effectively preserves more information by considering the individual importance of each KV cache.

llms efficiency

arXiv Machine Learning

Sep 15

Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

arXiv:2609.14507v1 Announce Type: cross Abstract: Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV...

By Enda Yu, Dezun Dong, Xiangke Liao

arXiv AI

Sep 16

Comparative Characterization of KV Cache Management Strategies for LLM Inference

arXiv:2604.05012v2 Announce Type: replace-cross Abstract: Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value...

By Oteo Mamo, Olga Kogiou, Hyunjin Yi, Weikuan Yu

llms efficiency benchmarks

One email a morning, machine-written

Related stories

CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

Comparative Characterization of KV Cache Management Strategies for LLM Inference