arXiv Computation and Language
TomasuLLM: Out-of-Order Speculative Execution for LLM Agents
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Jun 16
Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
arXiv:2603. 18897v2 Announce Type: replace-cross Abstract: LLM-powered agents execute tasks through a sequential loop of model generation and tool execution.
arXiv AI
Sep 4
Speculative Macro Commit for Faster Tool-Using Agents
Speculative Macro Commit (SMC) is a runtime technique for tool‑using language‑model agents that separates an authoritative actor model from a faster speculative drafter model. The drafter predicts and executes future action chains on a snapshot, storing recurring multi‑action patterns in a macro library. When the actor’s next tool call aligns with a drafted action, SMC commits the pre‑executed steps, reducing latency by up to 18.59% on certain benchmarks while maintaining accuracy.
Hugging Face Trending Papers
Sep 3
Speculative Macro Commit for Faster Tool-Using Agents
Speculative Macro Commit (SMC) is a runtime technique that speeds up tool‑using language‑model agents by having a fast speculative drafter model predict and execute future action chains on a separate environment snapshot. The drafter’s predictions are matched against a macro library of recurring multi‑action skeletons; when the authoritative actor’s next tool call aligns with the first drafted action, SMC commits the remaining pre‑executed steps to the official trajectory. Experiments with Qwen3.5 models show that SMC maintains overall accuracy while cutting latency by up to 18.6% on telecom benchmarks and 44.9% on AppWorld compared to sequential execution.
arXiv AI
Jul 28
SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving
arXiv:2607. 23933v1 Announce Type: cross Abstract: As LLM agents increasingly rely on the Model Context Protocol (MCP) to invoke isolated external sandboxes, disaggregated sandbox deployment introduces a fundamental tension between resource utilization and interactive tail latency.
arXiv Machine Learning
Aug 4
AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving
arXiv:2608. 00881v1 Announce Type: new Abstract: Large language model agents increasingly act through stateful tools, yet model generation and environment execution remain serialized at every step.
arXiv Machine Learning
Jul 10
Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems
arXiv:2607. 08010v1 Announce Type: cross Abstract: Production LLM agents often waste latency and reliability by regenerating code for the same procedural steps on every request.