← Back to all news
arXiv Machine Learning By Tejinder Singh, John Pflueger, Jeebak Mitra, Robert Lincourt, Mitchell Markow, Bhavesh A. Patel

Optimizing AI Inference Across the Deployment Stack

Read the original on arXiv Machine Learning →

The paper argues that AI deployment performance depends on interactions among compression, compiler transformations, and serving policies rather than just model architecture. It introduces a three‑layer taxonomy—model‑level techniques, compiler transformations, and system policies—and frames deployment as a constrained multi‑objective optimization problem over accuracy, latency, throughput, memory footprint, and energy. The authors propose an evidence protocol for comparable benchmarking and synthesize data from edge and data‑center platforms to show that cross‑layer interactions drive deployment outcomes, concluding with a constraint‑aware selection procedure and open research problems.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 28

A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models

The paper introduces a composite metric for quantizing small language models that balances information retention and throughput gains, using a normalized SQNR-based coefficient and roofline-based latency analysis. Profiling Gemma 3 1B shows that Feed‑Forward Network blocks and the embedding matrix are prime candidates for acceleration, with the metric enabling tuning of speed‑quality trade‑offs without actual execution. The authors demonstrate that their estimates predict accelerated speedup within about 4% error and allocate resources more effectively than evolutionary search or Shapley‑value methods.

By Artem Safronov
llmsragefficiency
More like this →
arXiv AI
Jun 30

KernelSight-LM: A Kernel-Level LLM Inference Simulator

arXiv:2606. 28565v1 Announce Type: cross Abstract: As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets.

By Xiteng Yao, Taeho Kim, Hengzhi Pei, Xinle Liu, Kyle Ulrich, Leonard Lausen, Ashish Khetan, Xiang Song, George Karypis, Martin Herbordt
llmsefficiencybenchmarks
More like this →
Hugging Face Trending Papers
Aug 27

A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models

The paper introduces a composite metric for quantizing small language models that balances information retention and throughput gains. By profiling Gemma 3 1B, it identifies Feed‑Forward Network blocks and the embedding matrix as key acceleration targets, estimating quality and speed scores without actual execution. The metric, validated across architectures with ~4% prediction error, enables predictable, resource‑efficient quantization compared to evolutionary or Shapley‑based methods.

llmsragefficiency
More like this →
arXiv Machine Learning
Jul 8

Leveraging Neural Graph Compilers in Machine Learning Research for Edge-Cloud Systems

arXiv:2504. 20198v2 Announce Type: replace-cross Abstract: This work presents a comprehensive evaluation of neural network graph compilers across heterogeneous hardware platforms, addressing the critical gap between theoretical optimization techniques and practical deployment scenarios.

By Alireza Furutanpey, Carmen Walser, Philipp Raith, Pantelis A. Frangoudis, Schahram Dustdar
efficiency
More like this →