arXiv AI By Chaoqian Ouyang, Ling Yue, Libin Zheng, Hanghui Guo, Shengxiang Xu, YiShu Wang, Ran Li, Jian Yin, Shaowu Pan, Shimin Di

TokenCast: Forecasting Token Consumption During LLM Agent Execution

Read the original on arXiv AI →

The Flow has not summarised this story yet — read it at arXiv AI.

arXiv AI
Oct 5

What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Compute

The paper introduces a Mixture-of-Agents (MoA) approach to quantify the per-token compute required by large language models. By having a panel of fifteen models of varying sizes attempt to reproduce each token, the authors define the smallest successful agent’s inference cost as the token’s sufficient compute, providing an upper bound on necessary computation. Experiments on benchmarks show that a 0.5B model can reproduce most tokens, and that the MoA-derived compute map can reduce latency in model routing and drafting tasks while improving or maintaining accuracy.

By Zhixu Du, Weijia Han, Hai Helen Li, Yiran Chen
arXiv AI
Oct 5

How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks

The paper presents a systematic study of token consumption in AI agents performing coding tasks. It finds that agentic tasks are far more expensive—about 1000 times more tokens than code reasoning or chat—primarily due to input tokens, and that token usage varies wildly, with accuracy peaking at moderate costs. Models differ significantly in efficiency, and current frontier models cannot reliably predict their own token usage, often underestimating it.

By Longju Bai, Zhemin Huang, Xingyao Wang, Jiao Sun, Rada Mihalcea, Erik Brynjolfsson, Alex Pentland, Jiaxin Pei
arXiv Machine Learning
Jul 17

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

arXiv:2607. 14952v1 Announce Type: new Abstract: A growing gap separates inference context lengths from RL post-training: inference systems are approaching million-token contexts, while post-training workloads often remain at 256K tokens or below and rely on length generalization at deployment.

By Changhai Zhou, Kieran Liu, Yuhua Zhou, Qian Qiao, Jun Gao, Harry Zhang, Irvine Lu, Nolan Ho, Lucian Li, Andrew Lei, Cleon Cheng, Steven Chiang, Yihang Zeng, Di Zhang, Rio Yang, Kaijie Chen, Andrew Chen, Pony Ma, Weizhong Zhang, Cheng Jin
Hugging Face Trending Papers
Nov 10

ForeSpark: Joint Draft and Verification Budget Scheduling for Parallel Speculative Decoding

ForeSpark introduces a work‑conserving speculation (WCS) framework that jointly schedules draft and verification budgets for parallel speculative decoding in large language model serving. It uses a context‑aware pre‑draft estimator, a lightweight joint scheduler, and a post‑draft budget allocator to reduce wasted compute and improve throughput. Experiments on vLLM show up to 1.77× higher throughput than DSpark and 2.49× over autoregressive decoding, while maintaining the same output distribution and supporting higher request rates under latency SLOs.