arXiv:2606. 18431v1 Announce Type: new Abstract: LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice.
By Yueying Li, Yuanfan Chen, Jiayang Chen, Esha Choukse, Haoran Qiu, G. Edward Suh, Rodrigo Fonseca, Ziv Scully, Udit Gupta
arXiv:2510. 03243v3 Announce Type: replace-cross Abstract: Efficient scheduling of large language model (LLM) inference tasks is critical for achieving low latency and high throughput, a challenge that is becoming increasingly acute with the rise of reasoning-capable LLMs whose generation lengths are highly variable.
By Yiheng Tao, Yihe Zhang, Matthew Dearing, Xin Wang, Yuping Fan, Michael E. Papka, Zhiling Lan
TIDE (Temporal Incremental Draft Engine) is a serving‑engine‑native framework that integrates online draft adaptation into high‑performance LLM inference. By reusing intermediate hidden states from the target model as training signals, TIDE avoids extra target model computation and serving‑time overhead, activating speculation and draft training only when beneficial. On heterogeneous GPU clusters, TIDE achieves up to 1.66× higher throughput than no‑speculation baselines, reduces training time by up to 3.02×, cuts storage needs by 24×, and improves system throughput by up to 1.22×.
By Jiyoung Park, Hankyu Jang, Changseok Song, Wookeun Jung
The paper introduces JIL, an attack that manipulates length predictions used by large language model (LLM) schedulers to gain higher priority and reduce completion time. By optimizing an adversarial suffix, JIL causes lightweight output-length probes to underestimate request lengths, leading to up to an 83.4 % reduction in predicted length and an average 1.53× faster completion in end‑to‑end serving experiments. The study also evaluates defenses, showing that grouping predictions into coarse intervals can mitigate JIL’s advantage and protect benign requests.
By Yuyang Dai, Rana Shahout, Mahmood Sharif
arXiv:2605. 04215v3 Announce Type: replace-cross Abstract: Diffusion-based Large Language Models (D-LLMs) represent a promising frontier in generative AI, offering fully parallel token generation that can lead to significant throughput advantages and superior GPU utilization over the traditional autoregressive paradigm.
By Michael Rottoli, Subhankar Roy, Stefano Paraboschi
arXiv:2608. 10362v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to speculate multiple tokens, reducing expensive target model decoding steps.
By Eunjeong Kim, Yeong Jun Jeon, Myeonggyun Han