arXiv Machine Learning October 23, 2026 By Kyrylo Yemets, Andrii Yaroshevych, Victor Muryn, Ostap Khomenko, Pavlo Kryven, Maksym Kmet, Maksym Shamrai, Mariya Hirna

Evaluating the Applicability of LLM Inference Optimizations on Apple Silicon

Read the original on arXiv Machine Learning →

The paper investigates how well common large‑language‑model (LLM) inference optimizations perform on Apple Silicon compared to server‑class NVIDIA GPUs. Four optimization families—KV‑cache pruning, speculative decoding, ported GPU kernels, and post‑training quantization—are evaluated on four Apple machines under a single‑request decode workload, with matched RTX 4090 comparisons where possible. Results show that KV‑cache pruning can boost Apple throughput up to 1.75×, while speculative decoding actually reduces throughput on Apple; CUDA kernel designs transfer only partially, and quantization gains vary by machine, indicating that speedups depend on the specific technique, runtime, hardware, and workload regime.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 9

LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load

arXiv:2603. 23640v2 Announce Type: replace-cross Abstract: Deploying large language models on-device for always-on personal agents demands sustained inference from hardware tightly constrained in power, thermal envelope, and memory.

By Pranay Tummalapalli, Sahil Arayakandy, Ritam Pal, Kautuk Kundan

llms agents efficiency benchmarks

arXiv AI

Jul 2

BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

arXiv:2607. 00501v1 Announce Type: cross Abstract: We present BaseRT, a native Metal inference runtime for large language models (LLMs) on Apple Silicon, and report the highest inference throughput on this hardware to date.

By Prabod Rathnayaka, Fabian Waschkowski, Lukas Wesemann

llms efficiency safety

arXiv Machine Learning

Aug 21

RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference

arXiv:2605. 23057v2 Announce Type: replace Abstract: RequestRouter is a lightweight request-boundary controller for reducing the latency and energy cost of single-GPU large language model inference.

By Aman Sunesh, Ali Alshehhi, Hivansh Dhakne

llms efficiency benchmarks

arXiv Machine Learning

Aug 4

Meganeura: Portable GPU Training and Inference through Vulkan and Metal

arXiv:2608. 01563v1 Announce Type: new Abstract: Training and deployed inference often cross export, conversion, and platform-specific runtime boundaries.

By Dzmitry Malyshau

arXiv AI

Sep 1

GreenBench: Benchmarking Energy Efficiency and Carbon Footprint of Open-Source LLM Inference on Apple Silicon

arXiv:2608.28667v1 Announce Type: cross Abstract: The rapid proliferation of Large Language Models (LLMs) has raised concerns about their environmental impact during inference. While Green AI researc...

By Rajeswari Kannan, Raj Firke, Shreya Bengle, Srushti Deshmukh

llms nlp efficiency benchmarks

arXiv AI

Sep 28

Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware

The paper reports a reproducible GPU power benchmark for 18 open‑source LLMs (0.5B–7B parameters) run on a single consumer RTX 4060ti GPU using the Ollama inference engine. Energy metrics such as mean/peak power, total energy per prompt, energy per output token, and throughput were measured, revealing that model architecture and quantization strategy, rather than parameter count alone, drive energy efficiency. The most efficient models were qwen2.5:0.5b and tinyllama:1.1b, while the 7B‑Mistral model consumed up to 8.6× more energy per token, and qwen3.5:0.8b(on) showed unusually high per‑prompt energy due to extended internal reasoning.

By Philipp M. Z\"ahl, Elja Dalipaj, Anika Hennig, Timon Bayer

llms efficiency benchmarks safety

One email a morning, machine-written

Related stories

LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load

BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference

Meganeura: Portable GPU Training and Inference through Vulkan and Metal

GreenBench: Benchmarking Energy Efficiency and Carbon Footprint of Open-Source LLM Inference on Apple Silicon

Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware