← Back to all news
arXiv AI By Joon Ha Kim, Geon-Woo Kim, Anoop Rachakonda, Daehyeok Kim

Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation

Read the original on arXiv AI →

Dooly is a profiling tool for large language model (LLM) inference that eliminates the need to re‑profile every operation for each configuration. By labeling input dimensions with taint propagation and only profiling operations missing from its latency database, Dooly builds a configuration‑agnostic, redundancy‑aware latency database that can be used as a drop‑in backend for simulators. In experiments across two GPU platforms, three attention backends, and multiple model architectures, Dooly achieves simulation accuracy within 5% MAPE for time‑to‑first‑token and 8% for total‑processing‑time while cutting profiling GPU‑hours by 56.4% compared to traditional methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jun 30

KernelSight-LM: A Kernel-Level LLM Inference Simulator

arXiv:2606. 28565v1 Announce Type: cross Abstract: As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets.

By Xiteng Yao, Taeho Kim, Hengzhi Pei, Xinle Liu, Kyle Ulrich, Leonard Lausen, Ashish Khetan, Xiang Song, George Karypis, Martin Herbordt
llmsefficiencybenchmarks
More like this →
arXiv AI
Jul 24

Profiling Lightweight Large Language Models

arXiv:2607. 20806v1 Announce Type: new Abstract: Lightweight large language models (LLMs) are increasingly being deployed locally on personal computers and are expected to play a growing role in resource-constrained edge and mobile environments.

By Tomohiro Harada, Enrique Alba, Gabriel Luque
llmsbenchmarks
More like this →
arXiv Machine Learning
Sep 14

HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications

HoliBench is a modular benchmarking and deployment toolkit that jointly measures accuracy, latency, and energy for foundation models across a wide range of devices, from single-board computers to GPU servers. It provides a platform abstraction layer that calibrates cross-device measurements and supports multiple model modalities, inference engines, and quantization levels. Using HoliBench, the authors evaluated 20 models on 7 device types, revealing tradeoffs such as limited latency gains from quantization on low‑precision hardware and diminishing accuracy returns relative to energy consumption, while also showing that single-model profiles can predict multi-model pipeline performance within a few percent.

By Inesh Chakrabarti, Zejun Xiong, Pragya Sharma, Mani Srivastava
llmsefficiencymultimodalbenchmarks
More like this →
arXiv AI
Aug 18

From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems

arXiv:2608. 15127v1 Announce Type: cross Abstract: Agentic applications are shifting AI serving from isolated model inference to long-running workloads in which LLMs coordinate tools, environments, and persistent state.

By Chaokun Chang, Yukun Zhou, Kaihua Fu, Dakai An, Tianyu Feng, Hanfeng Lu, Sheng Yao, Pu Guo, Yinghao Yu, Yizhou Shan, Bo Li, Binhang Yuan, Wei Wang
llmsagentsbenchmarks
More like this →