← Back to all news
arXiv Machine Learning By Artem Safronov

A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models

Read the original on arXiv Machine Learning →

The paper introduces a composite metric for quantizing small language models that balances information retention and throughput gains, using a normalized SQNR-based coefficient and roofline-based latency analysis. Profiling Gemma 3 1B shows that Feed‑Forward Network blocks and the embedding matrix are prime candidates for acceleration, with the metric enabling tuning of speed‑quality trade‑offs without actual execution. The authors demonstrate that their estimates predict accelerated speedup within about 4% error and allocate resources more effectively than evolutionary search or Shapley‑value methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

Hugging Face Trending Papers
Aug 27

A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models

The paper introduces a composite metric for quantizing small language models that balances information retention and throughput gains. By profiling Gemma 3 1B, it identifies Feed‑Forward Network blocks and the embedding matrix as key acceleration targets, estimating quality and speed scores without actual execution. The metric, validated across architectures with ~4% prediction error, enables predictable, resource‑efficient quantization compared to evolutionary or Shapley‑based methods.

llmsragefficiency
More like this →
arXiv Machine Learning
1d ago

Optimizing AI Inference Across the Deployment Stack

The paper argues that AI deployment performance depends on interactions among compression, compiler transformations, and serving policies rather than just model architecture. It introduces a three‑layer taxonomy—model‑level techniques, compiler transformations, and system policies—and frames deployment as a constrained multi‑objective optimization problem over accuracy, latency, throughput, memory footprint, and energy. The authors propose an evidence protocol for comparable benchmarking and synthesize data from edge and data‑center platforms to show that cross‑layer interactions drive deployment outcomes, concluding with a constraint‑aware selection procedure and open research problems.

By Tejinder Singh, John Pflueger, Jeebak Mitra, Robert Lincourt, Mitchell Markow, Bhavesh A. Patel
llmsefficiencybenchmarks
More like this →