Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets
Read the original on arXiv AI →The paper investigates how speculative decoding—where a draft model proposes multiple future tokens that are then verified by the target model—affects energy consumption in large language model serving under latency constraints. It finds that energy savings depend heavily on the acceptance rate of drafted tokens, which varies by model architecture, drafting scheme, and the type of text (e.g., conversation vs. code, English vs. Romansh). The study shows that while speculation can reduce energy per token at small batch sizes, at larger batches the verification cost dominates, potentially turning speculation into an energy loss, especially when acceptance rates are low.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.