arXiv AI
TokenCast: Forecasting Token Consumption During LLM Agent Execution
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jul 17
LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget
arXiv:2607. 14952v1 Announce Type: new Abstract: A growing gap separates inference context lengths from RL post-training: inference systems are approaching million-token contexts, while post-training workloads often remain at 256K tokens or below and rely on length generalization at deployment.
arXiv AI
Sep 23
Total Cost of Agency: Exact Attribution of Memory Injection Cost in Multi-Agent LLM Workflows
arXiv:2609.23790v1 Announce Type: new Abstract: Every node in a multi-agent large language model (LLM) workflow retrieves context from memory and injects it into its prompt, where those injected toke...
arXiv Machine Learning
Aug 4
When Replanning Becomes the Bottleneck: Budgeted Replanning for Embodied Agents
arXiv:2608. 01428v1 Announce Type: cross Abstract: Embodied agents replan frequently to recover from execution drift, partial observability, and coordination hazards, but each LLM-based replanning call can consume an accumulated textual context that grows over time and across agents.
arXiv AI
Aug 18
When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction
arXiv:2608. 15592v1 Announce Type: new Abstract: Efficient LLM serving is often bottlenecked by the need to pad sequences to a fixed maximum length, and this wastes compute and degrades throughput.
arXiv AI
Sep 4
Learning What Not to Forget: Long-Horizon Agent Memory from a Few Kilobytes of Learning
The paper introduces LRE (Learned Relevance Eviction), a lightweight, CPU‑only, language‑model‑free scorer that learns which parts of an agent’s interaction history are task‑critical and preserves them verbatim. In experiments, LRE matches or surpasses baseline eviction policies on accuracy‑cost trade‑offs, recovers 93% of full‑history accuracy, reduces worst‑case prompt size by 52%, and outperforms dense and token‑pruning encoders in conversational memory while being 295–1569× smaller. The method also achieves superior budgeted answer quality on LoCoMo reading and can be trained annotation‑free, recovering 95% of supervised scorer performance.
arXiv Machine Learning
Jul 15
Speculate with Memory: Lossless Acceleration for LLM Agents
arXiv:2607. 12236v1 Announce Type: new Abstract: Speculative execution accelerates LLM agents by using a smaller, cheaper model to predict and pre-launch the next step while the environment is idle.