arXiv AI

Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

arXiv:2606. 17104v1 Announce Type: cross Abstract: As large language models (LLMs) are increasingly deployed in latency- and cost-sensitive settings, inference efficiency has become a central systems challenge.

arXiv Machine Learning

Jul 14

FastTPS: An Optimized Method for LLM Token Phase for AI accelerators

arXiv:2607. 11211v1 Announce Type: new Abstract: The popularity of large language models (LLMs) escalates an ongoing demand for effective inference.

By Wenzong Yang, Danyang Zhang, Kun Cao, Tejus Siddagangaiah, Rajeev Patwari, Zhanxing Pu, Siyin Kong, Zijiang Yang, Hao Zhu, Varun Sharma, Yue Gao, Tianping Li, Fan Yang, Jicheng Chen, Yushan Chen, Fennian Zhao, Aaron Ng, Elliott Delaye, Ashish Sirasao, Sudip Nag

llms efficiency

arXiv AI

Jul 21

KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?

arXiv:2607. 16241v1 Announce Type: cross Abstract: Recent large language models (LLMs) can generate custom CUDA kernels that appear to outperform PyTorch on benchmarks such as KernelBench.

By Yunxiang Zhang (Xiangjun), Ping Yu (Xiangjun), Jianyu Wang (Xiangjun), Max (Xiangjun), Fan, Julian Reed, Azalia Mirhoseini, Will Su

llms benchmarks

arXiv AI

Jun 2

Threshold-Based Exclusive Batching for LLM Inference

arXiv:2606. 00516v1 Announce Type: new Abstract: Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization.

By Weifang Zhang, Yuzhou Nie, Bowen Pang, Guangrui Ma, Shining Wu

llms efficiency

arXiv Machine Learning

Jul 8

Performance Optimization and Comparative Analysis of Generative AI Models on Advanced Accelerators

arXiv:2607. 05400v1 Announce Type: cross Abstract: Generative AI models, such as Large Language Models (LLMs) and diffusion models, have demonstrated impressive performance across a wide range of tasks.

By Amitash Nanda, Javier Hernandez Nicolau, Madhusudan Gujral, Mahidhar Tatineni, Amitava Majumdar, Debashis Sahoo

llms diffusion fine-tuning efficiency

arXiv AI

Jun 30

KernelSight-LM: A Kernel-Level LLM Inference Simulator

arXiv:2606. 28565v1 Announce Type: cross Abstract: As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets.

By Xiteng Yao, Taeho Kim, Hengzhi Pei, Xinle Liu, Kyle Ulrich, Leonard Lausen, Ashish Khetan, Xiang Song, George Karypis, Martin Herbordt

llms efficiency benchmarks

arXiv AI

Aug 11

LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving

arXiv:2608. 08382v1 Announce Type: new Abstract: As LLM inference shifts to multi-tenant GPU clusters, co-batching improves throughput but obscures per-tenant usage and limits control.

By Shuowei Jin, Xueshen Liu, Jiaxin Shan, Le Xu, Tieying Zhang, Liguang Xie, Z. Morley Mao

llms efficiency

arXiv Machine Learning

Jul 31

KernelGenBench: A Multi-Source and Multi-Chip Benchmark for LLM-based Kernel Generation

arXiv:2607. 27231v1 Announce Type: cross Abstract: Large language models (LLMs) have significantly increased the demand for efficient accelerator kernels, but kernel development remains a highly specialized and labor-intensive task.

By Peiyu Zang, Jian Tao, Jialing Zhang, Yichen Yuan, Wentao Zhang, Guang Liu, Yonghua Lin

llms agents benchmarks

arXiv AI

Jun 24

VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing

arXiv:2509. 04827v3 Announce Type: replace-cross Abstract: The energy cost of Large Language Model (LLM) inference is rapidly becoming a barrier to sustainable and scalable deployment.

By Jiahuan Yu, Aryan Taneja, Junfeng Lin, Minjia Zhang

llms

arXiv AI

Jun 2

Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference

arXiv:2603. 29002v3 Announce Type: replace-cross Abstract: Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning.

By Zifan He, Rui Ma, Yizhou Sun, Jason Cong

llms rag

arXiv Machine Learning

Jun 2

DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing

arXiv:2511. 04791v2 Announce Type: replace Abstract: Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases.

By Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong, Mark Hill, Murali Annavaram

llms efficiency benchmarks

arXiv AI

Jul 28

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

arXiv:2607. 24434v1 Announce Type: cross Abstract: Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory.

By Dengke Han

llms efficiency

arXiv AI

Jun 16

Communication-Efficient Verifiable Attention for LLM Inference

arXiv:2606. 16352v1 Announce Type: cross Abstract: Computation integrity of remote large language model (LLM) serving can be questionable.

By Ziqun Chen, Ming Wu, Michael Heinrich, Jason Zeng, Huiying Lan, Tianwei Zhang, Rui Tan

llms

Related stories