arXiv Machine Learning
Jun 18
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
arXiv:2606. 18431v1 Announce Type: new Abstract: LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice.
LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction
LEAP is a length prediction system for large language model inference that uses the serving model’s prefill hidden states and a linear readout to estimate output length. It adapts online via recursive least squares, updating the readout with completed requests without costly retraining or gradient computation. Experiments show LEAP maintains strong ranking performance across domain shifts, improves Spearman correlation by 64.7% under large shifts, and reduces per-token latency by 15.9–40.8% while increasing throughput and drastically cutting predictor parameters.
Related stories
arXiv AI
Jun 29
Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank
arXiv:2510. 03243v3 Announce Type: replace-cross Abstract: Efficient scheduling of large language model (LLM) inference tasks is critical for achieving low latency and high throughput, a challenge that is becoming increasingly acute with the rise of reasoning-capable LLMs whose generation lengths are highly variable.
arXiv AI
Sep 25
TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference
TIDE (Temporal Incremental Draft Engine) is a serving‑engine‑native framework that integrates online draft adaptation into high‑performance LLM inference. By reusing intermediate hidden states from the target model as training signals, TIDE avoids extra target model computation and serving‑time overhead, activating speculation and draft training only when beneficial. On heterogeneous GPU clusters, TIDE achieves up to 1.66× higher throughput than no‑speculation baselines, reduces training time by up to 3.02×, cuts storage needs by 24×, and improves system throughput by up to 1.22×.
arXiv AI
Oct 5
Jumping the Line: Exploiting Length Predictions in LLM Scheduling
The paper introduces JIL, an attack that manipulates length predictions used by large language model (LLM) schedulers to gain higher priority and reduce completion time. By optimizing an adversarial suffix, JIL causes lightweight output-length probes to underestimate request lengths, leading to up to an 83.4 % reduction in predicted length and an average 1.53× faster completion in end‑to‑end serving experiments. The study also evaluates defenses, showing that grouping predictions into coarse intervals can mitigate JIL’s advantage and protect benign requests.
arXiv AI
Aug 3
Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs
arXiv:2605. 04215v3 Announce Type: replace-cross Abstract: Diffusion-based Large Language Models (D-LLMs) represent a promising frontier in generative AI, offering fully parallel token generation that can lead to significant throughput advantages and superior GPU utilization over the traditional autoregressive paradigm.
arXiv AI
Aug 12
MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices
arXiv:2608. 10362v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to speculate multiple tokens, reducing expensive target model decoding steps.
arXiv Machine Learning
Sep 30
Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
arXiv:2609.38090v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models are a compelling architecture for scaling model capacity, making them especially attractive for deployment on resource-...
arXiv AI
Jun 30
KernelSight-LM: A Kernel-Level LLM Inference Simulator
arXiv:2606. 28565v1 Announce Type: cross Abstract: As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets.
arXiv Machine Learning
Sep 10
MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
arXiv:2609.07966v1 Announce Type: new Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-cont...
arXiv AI
Oct 16
Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools
The paper introduces BEA, a fine‑grained estimator that predicts the time‑to‑first‑token (TTFT) for language model queries by replaying token‑batch formation based on current workload snapshots. BEA achieves about 5% error with 0.1 ms prediction time, enabling the latency‑aware router BEAR to balance accuracy, cost, and latency across heterogeneous LLM pools. BEAR outperforms seven baselines, improving utility of on‑target responses by 63% on average and by 30.5% at each arrival rate.
arXiv AI
Aug 18
When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction
arXiv:2608. 15592v1 Announce Type: new Abstract: Efficient LLM serving is often bottlenecked by the need to pad sequences to a fixed maximum length, and this wastes compute and degrades throughput.
arXiv AI
Oct 15
Stable Token-level Scheduling for Serverless LLM Inference
arXiv:2610.17689v1 Announce Type: new Abstract: Serverless inference is emerging as an efficient deployment model for Large Language Models (LLMs) due to its ability to elastically scale resources, i...