How Long Prompts Block Other Requests - Optimizing LLM Performance

Efficient Request Queueing – Optimizing LLM Performance

Prefill and Decode for Concurrent Requests - Optimizing LLM Performance

Optimizing your LLM in production

Multi-Bin Batching for Increasing LLM Inference Throughput

arXiv:2412. 04504v2 Announce Type: replace-cross Abstract: As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical.

LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving

arXiv:2608. 08382v1 Announce Type: new Abstract: As LLM inference shifts to multi-tenant GPU clusters, co-batching improves throughput but obscures per-tenant usage and limits control.

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.

LLM within MCP Matters: Measuring Inefficient Resource Utilization Driven by LLMs

arXiv:2608. 08467v1 Announce Type: new Abstract: The Model Context Protocol (MCP) standardizes how servers expose data and tools to Large Language Models (LLMs).

LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm

arXiv:2608. 06135v1 Announce Type: new Abstract: Large Language Models (LLMs) such as ChatGPT and Claude are widely used for information retrieval and problem-solving.

Optimum-NVIDIA Unlocking blazingly fast LLM inference in just 1 line of code

RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference

arXiv:2605. 23057v2 Announce Type: replace Abstract: RequestRouter is a lightweight request-boundary controller for reducing the latency and energy cost of single-GPU large language model inference.

Long Context Isn’t Free — I Built a Safe Prompt-Pruning Layer That Makes LLM Systems Work

LLMs don’t fail because they forget—they fail because they remember too much. As conversations grow, prompts accumulate redundant and low-value tokens, driving up cost and latency while silently degrading output quality.

Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model

Enterprise Document Intelligence [Vol. 1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right.