The paper reports a reproducible GPU power benchmark for 18 open‑source LLMs (0.5B–7B parameters) run on a single consumer RTX 4060ti GPU using the Ollama inference engine. Energy metrics such as mean/peak power, total energy per prompt, energy per output token, and throughput were measured, revealing that model architecture and quantization strategy, rather than parameter count alone, drive energy efficiency. The most efficient models were qwen2.5:0.5b and tinyllama:1.1b, while the 7B‑Mistral model consumed up to 8.6× more energy per token, and qwen3.5:0.8b(on) showed unusually high per‑prompt energy due to extended internal reasoning.
By Philipp M. Z\"ahl, Elja Dalipaj, Anika Hennig, Timon Bayer
arXiv:2610.17531v1 Announce Type: cross
Abstract: While it is known that quantization of large language models (LLMs) reduces memory usage via lower-bit weights, studies quantifying the resulting imp...
By Petar Basta
arXiv:2607. 05475v1 Announce Type: cross Abstract: Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency.
By Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren, Jinliang Yuan, Lingkun Li, Jiliang Wang
The operational energy consumption of large language model (LLM) inference is becoming an increasingly important component of the environmental footprint of deployed AI systems. However, direct measurement of inference energy often requires hardware telemetry, power instrumentation, or infrastructure-specific monitoring, limiting its applicability in comparative studies, early-stage system design, and sustainability reporting.
arXiv:2603. 23640v2 Announce Type: replace-cross Abstract: Deploying large language models on-device for always-on personal agents demands sustained inference from hardware tightly constrained in power, thermal envelope, and memory.
By Pranay Tummalapalli, Sahil Arayakandy, Ritam Pal, Kautuk Kundan
arXiv:2607. 26571v1 Announce Type: new Abstract: The operational energy consumption of large language model (LLM) inference is becoming an increasingly important component of the environmental footprint of deployed AI systems.
By Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis, George Dasoulas, Michael Keckeisen, Konstantinos Skianis, Sotirios Kotsopoulos, Francesca Dominici
arXiv:2606. 23001v1 Announce Type: cross Abstract: On-device LLM inference is increasingly attractive for privacy-preserving, reliable, and cost-effective deployment, yet its energy and thermal costs remain a critical bottleneck.
By Bohua Zou, Nian Liu, Binqi Sun, Matteo Mascherin, Debayan Roy, Yutao Liu, Yu Peng, Ning Jia, Haibo Chen
The paper investigates how well common large‑language‑model (LLM) inference optimizations perform on Apple Silicon compared to server‑class NVIDIA GPUs. Four optimization families—KV‑cache pruning, speculative decoding, ported GPU kernels, and post‑training quantization—are evaluated on four Apple machines under a single‑request decode workload, with matched RTX 4090 comparisons where possible. Results show that KV‑cache pruning can boost Apple throughput up to 1.75×, while speculative decoding actually reduces throughput on Apple; CUDA kernel designs transfer only partially, and quantization gains vary by machine, indicating that speedups depend on the specific technique, runtime, hardware, and workload regime.
By Kyrylo Yemets, Andrii Yaroshevych, Victor Muryn, Ostap Khomenko, Pavlo Kryven, Maksym Kmet, Maksym Shamrai, Mariya Hirna
arXiv:2610.14389v1 Announce Type: new
Abstract: Benchmark scores tell us what an LLM can do, but not how much energy is required to deliver that capability. We directly measure this trade-off for 86...
By Zeyu Yang, Wes Armour
arXiv:2607. 00501v1 Announce Type: cross Abstract: We present BaseRT, a native Metal inference runtime for large language models (LLMs) on Apple Silicon, and report the highest inference throughput on this hardware to date.
By Prabod Rathnayaka, Fabian Waschkowski, Lukas Wesemann
Large Language Models (LLMs) are increasingly used to optimize code for performance, but this study shows that such optimizations often increase energy consumption rather than reduce it. Across 11 open‑source models and 1,000 benchmark problems, inference energy ranged from 0.79 to 8.68 MJ, and 42 of 44 configurations produced code that used 4% to 886% more energy than human‑written reference solutions. Mixture‑of‑Experts architectures and token‑limit constraints can reduce inference energy, yet performance‑optimized code remains energy‑heavier, indicating that speed and sustainability are not always aligned goals.
By Sadra Emami, Benjamin Weigell, Mohammad Reza Mousavi, Gunel Jahangirova, Hugo Araujo, Bernhard Bauer
Large Language Model (LLM) inference workloads are a rapidly growing contributor to data center energy consumption. Optimizing these deployments requires matching specific LLMs to the most efficient GPUs, but operators currently lack the tools to do so without exhaustively profiling each combination.