← Back to all news
arXiv Machine Learning By Tejinder Singh, John Pflueger, Jeebak Mitra, Robert Lincourt, Mitchell Markow, Bhavesh A. Patel

Optimizing AI Inference Across the Deployment Stack

Read the original on arXiv Machine Learning →

The paper argues that AI deployment performance depends on interactions among compression, compiler transformations, and serving policies rather than just model architecture. It introduces a three‑layer taxonomy—model‑level techniques, compiler transformations, and system policies—and frames deployment as a constrained multi‑objective optimization problem over accuracy, latency, throughput, memory footprint, and energy. The authors propose an evidence protocol for comparable benchmarking and synthesize data from edge and data‑center platforms to show that cross‑layer interactions drive deployment outcomes, concluding with a constraint‑aware selection procedure and open research problems.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 14

HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications

HoliBench is a modular benchmarking and deployment toolkit that jointly measures accuracy, latency, and energy for foundation models across a wide range of devices, from single-board computers to GPU servers. It provides a platform abstraction layer that calibrates cross-device measurements and supports multiple model modalities, inference engines, and quantization levels. Using HoliBench, the authors evaluated 20 models on 7 device types, revealing tradeoffs such as limited latency gains from quantization on low‑precision hardware and diminishing accuracy returns relative to energy consumption, while also showing that single-model profiles can predict multi-model pipeline performance within a few percent.

By Inesh Chakrabarti, Zejun Xiong, Pragya Sharma, Mani Srivastava
llmsefficiencymultimodalbenchmarks
More like this →
Hugging Face Trending Papers
Sep 28

AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs

AgentPerfBench is a new benchmarking suite designed to evaluate the inference performance of agentic large language models (LLMs) that handle multi‑turn, tool‑using, and context‑expanding tasks. It builds on real traces from agentic benchmarks such as SWE‑Bench and TerminalBench, and generates synthetic profiles that reflect realistic input/output lengths and turn counts. The suite also provides kernel‑level Nsight Compute traces and a multi‑dimensional roofline model to identify hardware bottlenecks and quantify the gap between traditional chat benchmarks and agentic workloads.

llmsagentsbenchmarks
More like this →
arXiv AI
Oct 7

Agentic Design Space Exploration for Joint Hardware Configuration Selection and Mapping of AI Inference Workloads on Heterogeneous Edge SoCs

The paper introduces TraceDSE, an agentic design space exploration framework for jointly mapping AI inference workloads to heterogeneous edge SoCs and configuring each processing unit. Unlike traditional black-box optimization, TraceDSE uses a proposer‑critic loop powered by large language models and enriched with system execution traces to identify bottlenecks and refine design choices. Experiments on an Intel Meteor Lake SoC show that TraceDSE outperforms state‑of‑the‑art evolutionary and Bayesian methods, improving Pareto frontier hypervolume by up to 68% while reducing hardware evaluations by 6–9×.

By Geetha Prasuna Yarramneni, Surya Selvam, Wilfried Haensch, Anand Raghunathan
llmsagentsbenchmarks
More like this →