← Back to all news
arXiv Machine Learning By Mobina Kashaniyan, Ali Jannesari

Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling

Read the original on arXiv Machine Learning →

The paper demonstrates that the number of candidates generated during test-time scaling of large language models does not fully capture the system cost. By comparing different generation schedules (e.g., one batched call versus multiple serial calls) while keeping the total candidate count fixed, the authors show that serial calls consume significantly more GPU energy and latency. The study suggests that reporting candidate count alone is insufficient; evaluations should also include generation schedule and GPU-level metrics.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Sep 17

The Inference Engineering Pareto Atlas: Which Optimizations Dominate the Cost, Quality, and Latency Frontier?

The paper presents a Pareto atlas of LLM inference optimizations, mapping cost, quality, and latency trade‑offs for Qwen2.5‑7B‑Instruct on L4, A100, and H100 GPUs. Using 54 measured configurations and a calibrated simulator, it identifies 18 of 36 setups on the Pareto frontier, showing that combined methods outperform single ones. Quality tests reveal that AWQ 4bit and FP8 weights offer significant latency reductions while largely preserving accuracy, but naive FP8 KV caching fails to answer any questions correctly.

By Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly
llmsefficiency
More like this →
arXiv Machine Learning
Oct 3

How Much Can Language Models Gain from Test-Time Computation?

The paper investigates how test‑time computation can enhance language models and at what cost, introducing the SELF‑POT benchmark to evaluate this across competition mathematics, competitive programming, and agentic workflows. SELF‑POT separates candidate coverage from final accuracy, tracks correctness transitions under revision, and measures protocol completion alongside task success. Using a unified budget rule, the study compares direct inference, parallel sampling, and self‑revision across five low‑cost reasoning models, revealing that selection rules and failure handling significantly influence gains and cost savings.

By Bangji Yang, Jingyuan Li, Jiajun Fan, Yi Evie Zhang, Ruihan Guo, Hongba Ma, Neil He, Chumeng Liang, Qinglong Zheng, Zhanghan Ni, Ge Liu
llmsagentsbenchmarks
More like this →
arXiv AI
Aug 11

Thought-Level Beam Search for Reasoning

arXiv:2608. 08020v1 Announce Type: new Abstract: Test-time compute scaling is a primary driver of performance in large reasoning models (LRMs), but extreme inefficiency bounds current approaches, shifting the critical question from \emph{how much} compute to spend, to \emph{where} to allocate it.

By Lijie Yang, Hongyin Luo, Tri Dao, Ravi Netravali
efficiencybenchmarks
More like this →
arXiv AI
Oct 26

Green Code or Costly Code? Evaluating the Energy Costs of Iterative LLM-Based Code Optimizations

Large Language Models (LLMs) are increasingly used to optimize code for performance, but this study shows that such optimizations often increase energy consumption rather than reduce it. Across 11 open‑source models and 1,000 benchmark problems, inference energy ranged from 0.79 to 8.68 MJ, and 42 of 44 configurations produced code that used 4% to 886% more energy than human‑written reference solutions. Mixture‑of‑Experts architectures and token‑limit constraints can reduce inference energy, yet performance‑optimized code remains energy‑heavier, indicating that speed and sustainability are not always aligned goals.

By Sadra Emami, Benjamin Weigell, Mohammad Reza Mousavi, Gunel Jahangirova, Hugo Araujo, Bernhard Bauer
llmsbenchmarks
More like this →