arXiv AI June 2, 2026 By Weifang Zhang, Yuzhou Nie, Bowen Pang, Guangrui Ma, Shining Wu

Threshold-Based Exclusive Batching for LLM Inference

arXiv:2606. 00516v1 Announce Type: new Abstract: Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Aug 31

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

arXiv:2608.28044v1 Announce Type: cross Abstract: Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes t...

By Prabhu Vellaisamy, Vanessa Lam, Shawn Blanton, John Paul Shen

llms

arXiv AI

Jun 16

Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints

arXiv:2504. 11320v4 Announce Type: replace-cross Abstract: Large language models now serve millions of users daily, with providers incurring costs exceeding $700,000 per day.

By Ruicheng Ao, Gan Luo, David Simchi-Levi, Xinshang Wang

llms benchmarks safety

arXiv Machine Learning

Sep 15

BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference

arXiv:2609.13592v1 Announce Type: cross Abstract: GPU memory bandwidth and capacity limit throughput in large language model (LLM) inference. The GPU memory system consists of a primary tier of high-...

By Anish Saxena, Jae Hyung Ju, Hritvik Taneja, Po-An Tsai, Aamer Jaleel, Christos Kozyrakis, Moinuddin Qureshi

llms efficiency

arXiv AI

Jun 17

Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

arXiv:2606. 17104v1 Announce Type: cross Abstract: As large language models (LLMs) are increasingly deployed in latency- and cost-sensitive settings, inference efficiency has become a central systems challenge.

By Shun Usami, Venkatram Vishwanath, E. Wes Bethel

llms efficiency

arXiv AI

Jul 3

Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving

arXiv:2607. 02043v1 Announce Type: cross Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering.

By Shrikara Arun, Anjaly Parayil, Srikant Bharadwaj, Renee St. Amant, Victor R\"uhle

llms benchmarks

arXiv Machine Learning

Aug 18

Multi-Bin Batching for Increasing LLM Inference Throughput

arXiv:2412. 04504v2 Announce Type: replace-cross Abstract: As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical.

By Ozgur Guldogan, Jackson Kunde, Kangwook Lee, Ramtin Pedarsani

llms efficiency

One email a morning, machine-written

Related stories

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints

BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference

Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving

Multi-Bin Batching for Increasing LLM Inference Throughput