arXiv AI June 12, 2026 By Shan Yu, Yifan Qiao, Mingyuan Ma, Yangmin Li, Shuo Yang, Xinyuan Tong, Yang Wang, Zhiqiang Xie, Yuwei An, Shiyi Cao, Ke Bao, Deepak Vij, Xiaoning Ding, Yichen Wang, Qingda Lu, Zhong Wang, Gao Gao, Harry Xu, Junyi Shu, Jiarong Xing, Ying Sheng

Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning

Read the original on arXiv AI →

arXiv:2505. 04021v3 Announce Type: replace-cross Abstract: Inference providers must maintain availability for many LLMs, including low-volume but essential models, making resource efficiency increasingly important as token prices fall.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

arXiv AI

Sep 24

EMA: Elastic and Performance Transparent Memory Across GPUs

arXiv:2609.27040v1 Announce Type: cross Abstract: Multi-GPU servers have become the standard building block of modern data centers, providing aggregated capacity through high-bandwidth interconnects....

By Yi Xu, Tian Xia, Ion Stoica

llms efficiency

arXiv Machine Learning

Jun 5

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

arXiv:2606. 06302v1 Announce Type: new Abstract: Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth.

By Hyungmin Kim, Minsoo Kim, Hongseok Kim, Jungwook Choi

llms efficiency

arXiv AI

Jun 24

CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

arXiv:2606. 24506v1 Announce Type: cross Abstract: Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold.

By Zhuoren Ye, Tianyu Wo, Dinghao Xue, Mingming Zhang, Yuchen Teng, Chunming Hu, Renyu Yang

llms benchmarks

Hugging Face Trending Papers

Jun 4

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth. Non-uniform KV compression effectively preserves more information by considering the individual importance of each KV cache.

llms efficiency

arXiv AI

Jul 1

An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU

arXiv:2603. 16428v2 Announce Type: replace-cross Abstract: Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPUs.

By Ruijia Yang, Zeyi Wen

llms fine-tuning efficiency

arXiv AI

Aug 11

LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving

arXiv:2608. 08382v1 Announce Type: new Abstract: As LLM inference shifts to multi-tenant GPU clusters, co-batching improves throughput but obscures per-tenant usage and limits control.

By Shuowei Jin, Xueshen Liu, Jiaxin Shan, Le Xu, Tieying Zhang, Liguang Xie, Z. Morley Mao

llms efficiency

One email a morning, machine-written

Related stories

EMA: Elastic and Performance Transparent Memory Across GPUs

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU

LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving