arXiv AI
Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 8
AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
arXiv:2605. 08692v2 Announce Type: replace Abstract: Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference.
arXiv AI
Sep 1
GreenBench: Benchmarking Energy Efficiency and Carbon Footprint of Open-Source LLM Inference on Apple Silicon
arXiv:2608.28667v1 Announce Type: cross Abstract: The rapid proliferation of Large Language Models (LLMs) has raised concerns about their environmental impact during inference. While Green AI researc...
arXiv AI
Sep 28
Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware
The paper reports a reproducible GPU power benchmark for 18 open‑source LLMs (0.5B–7B parameters) run on a single consumer RTX 4060ti GPU using the Ollama inference engine. Energy metrics such as mean/peak power, total energy per prompt, energy per output token, and throughput were measured, revealing that model architecture and quantization strategy, rather than parameter count alone, drive energy efficiency. The most efficient models were qwen2.5:0.5b and tinyllama:1.1b, while the 7B‑Mistral model consumed up to 8.6× more energy per token, and qwen3.5:0.8b(on) showed unusually high per‑prompt energy due to extended internal reasoning.
arXiv Computation and Language
Sep 1
Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
The paper presents a budget‑aware compression pipeline for deploying 70B‑parameter language models on a single NVIDIA GPU. It examines how pruning, quantization, and KV‑cache compression interact, showing that layer‑wise pruning improves weight quantization robustness and that KV‑cache sparsification complements INT8 KV quantization without harming decoding speed. Using these insights, the authors compressed a 70B model to ~33 GB, achieving ~57 tokens/s on 10k‑token prompts on an A40 while maintaining accuracy within 5% on standard benchmarks.
Hugging Face Trending Papers
Jun 25
CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs. Unlike existing state-of-the-art ternary quantization methods that rely on data-intensive and costly quantization-aware training to mitigate severe performance degradation, CAT-Q is a simple yet effective post-training quantization scheme that is readily applicable to LLMs with diverse architectures and model sizes.
arXiv Machine Learning
Aug 31
Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms
arXiv:2608.28044v1 Announce Type: cross Abstract: Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes t...