The paper presents a Pareto atlas of LLM inference optimizations, mapping cost, quality, and latency trade‑offs for Qwen2.5‑7B‑Instruct on L4, A100, and H100 GPUs. Using 54 measured configurations and a calibrated simulator, it identifies 18 of 36 setups on the Pareto frontier, showing that combined methods outperform single ones. Quality tests reveal that AWQ 4bit and FP8 weights offer significant latency reductions while largely preserving accuracy, but naive FP8 KV caching fails to answer any questions correctly.
By Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly
The paper investigates how test‑time computation can enhance language models and at what cost, introducing the SELF‑POT benchmark to evaluate this across competition mathematics, competitive programming, and agentic workflows. SELF‑POT separates candidate coverage from final accuracy, tracks correctness transitions under revision, and measures protocol completion alongside task success. Using a unified budget rule, the study compares direct inference, parallel sampling, and self‑revision across five low‑cost reasoning models, revealing that selection rules and failure handling significantly influence gains and cost savings.
By Bangji Yang, Jingyuan Li, Jiajun Fan, Yi Evie Zhang, Ruihan Guo, Hongba Ma, Neil He, Chumeng Liang, Qinglong Zheng, Zhanghan Ni, Ge Liu
arXiv:2608. 08020v1 Announce Type: new Abstract: Test-time compute scaling is a primary driver of performance in large reasoning models (LRMs), but extreme inefficiency bounds current approaches, shifting the critical question from \emph{how much} compute to spend, to \emph{where} to allocate it.
By Lijie Yang, Hongyin Luo, Tri Dao, Ravi Netravali
Large Language Models (LLMs) are increasingly used to optimize code for performance, but this study shows that such optimizations often increase energy consumption rather than reduce it. Across 11 open‑source models and 1,000 benchmark problems, inference energy ranged from 0.79 to 8.68 MJ, and 42 of 44 configurations produced code that used 4% to 886% more energy than human‑written reference solutions. Mixture‑of‑Experts architectures and token‑limit constraints can reduce inference energy, yet performance‑optimized code remains energy‑heavier, indicating that speed and sustainability are not always aligned goals.
By Sadra Emami, Benjamin Weigell, Mohammad Reza Mousavi, Gunel Jahangirova, Hugo Araujo, Bernhard Bauer
arXiv:2610.14389v1 Announce Type: new
Abstract: Benchmark scores tell us what an LLM can do, but not how much energy is required to deliver that capability. We directly measure this trade-off for 86...
By Zeyu Yang, Wes Armour
arXiv:2610.07094v1 Announce Type: cross
Abstract: LLM deployment is shifting from single-turn completion to agentic trajectories in which a model plans, calls tools, reads results and reasons at test...
By Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona Zahid
The paper examines test‑time scaling (TTS) methods that use extra inference compute to improve language model outputs. Across five open‑ended benchmarks—medicine, law, finance, general chat, and creative writing—the study finds that increasing exploration (generating more candidates) consistently yields better top candidates, but exploitation (selecting the final output) remains weak due to poor reward‑model correlation. Only the Fusion approach, which synthesizes candidates, reliably improves results, yet it recovers only about 40% of the potential quality, indicating that the bottleneck lies in choosing from the candidate pool rather than generating it.
By Davide Romano, Kanak Raj, Jerrod Parker, Daniele Giofr\`e
arXiv:2610.19975v1 Announce Type: new
Abstract: Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonst...
By Zefeng He, Yunkai Chai, Xiaoyang Chen, Tong Zhu, Zongliang Wu, Yun Luo, Yafu Li, Yu Cheng
arXiv:2608.22048v1 Announce Type: new
Abstract: Large language models are increasingly deployed on local hardware for privacy, cost, and accessibility reasons. Yet many evaluations emphasize accuracy...
By Orion Powers, Daniella Seum, Khaled Slhoub
Parameter-efficient fine-tuning (PEFT) and low-bit quantization are now standard tools for adapting language models under tight compute budgets, yet their interaction is most often studied on billion-parameter models where the design space is expensive to explore. We ask a complementary question: on a specific, fully reproducible 60M-parameter encoder-decoder model (T5-small) and a single-table text-to-SQL benchmark (WikiSQL), how much task accuracy does each efficiency knob actually cost?
arXiv:2609.07966v1 Announce Type: new
Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-cont...
By Michael Wang, Keith Li, Roozbeh Bostandoost
arXiv:2607. 21453v1 Announce Type: new Abstract: Scaling inference-time computation has emerged as a reliable method to improve the performance of large language models on complex reasoning and programming tasks.
By Rajiv Shailesh Chitale, Rahul Madhavan, Taneesh Gupta, Deepanway Ghosal, Aravindan Raghuveer