arXiv AI

Total Cost of Agency: Exact Attribution of Memory Injection Cost in Multi-Agent LLM Workflows

arXiv AI
Oct 5

What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Compute

The paper introduces a Mixture-of-Agents (MoA) approach to quantify the per-token compute required by large language models. By having a panel of fifteen models of varying sizes attempt to reproduce each token, the authors define the smallest successful agent’s inference cost as the token’s sufficient compute, providing an upper bound on necessary computation. Experiments on benchmarks show that a 0.5B model can reproduce most tokens, and that the MoA-derived compute map can reduce latency in model routing and drafting tasks while improving or maintaining accuracy.

By Zhixu Du, Weijia Han, Hai Helen Li, Yiran Chen
arXiv Machine Learning
Jul 10

What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

arXiv:2607. 08032v1 Announce Type: new Abstract: Large language models, and the agents built on them, spend an ever-growing share of their compute and memory on remembering: caching attention keys and values, carrying long prompts, maintaining recurrent state, and storing what happened in previous turns and sessions.

By Ashwin Gerard Colaco, Nada Lahjouji
arXiv AI
Aug 24

Nexus: Depth-Adaptive KV-Cache Splicing and Retrieval-Decoupled Tool Routing for Agentic LLMs on Unified Memory

Nexus introduces a depth‑adaptive KV‑cache splicing and retrieval‑decoupled tool routing mechanism for agentic large language models that reduces the time‑to‑first‑token (TTFT) by decoupling tool routing from the expensive schema re‑encoding step. It uses an INT8 semantic lookaside buffer to select tools via retrieval and generates arguments from a compressed textual signature, maintaining about 89% routing accuracy even as the tool registry scales to 250 tools. Additionally, Nexus can splice compiled schema KV blocks into the live context, repairing the seam with a depth‑adaptive suffix redecode when rotary position embedding drift exceeds a threshold, ensuring output fidelity while achieving up to 1.7× TTFT speedup at moderate depth.

By Mustafa Arslan
arXiv AI
Oct 1

Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost

The paper introduces Galahad, a memory layer that stores a transformer language model’s key‑value state for blocks of text, allowing subsequent requests to reuse previously computed attention rather than recomputing it. On seven real‑world datasets, 98.7% of prompt tokens were already read, and with Galahad the model could attend to an entire 97,000‑token corpus, achieving 98–100% recall on a 100‑fact test while reducing inference time and energy consumption dramatically. The approach was validated across 30 models and all runtimes, demonstrating that stateful inference can replace stateless serving without loss of accuracy.

By Sietse Schelpe
arXiv AI
Oct 2

Pay for the Fault, Not the Flow: Label-Free In-Flow Multi-Agent Workflow Optimization

The paper introduces InFlowOp, a label‑free optimization framework that assigns costs to each decision in a multi‑agent workflow, balancing agent competence against execution time. It determines task granularity and agent assignment before execution and corrects faults during execution using the same cost metric. The authors also present Braid, a benchmark for multi‑agent coordination, and show that InFlowOp outperforms single‑agent baselines by up to 11.97% across various domains.

By Xuehang Guo, Haoyu Wang, Shengyu Chen, Zach Chen, Wei Cheng, Qingyun Wang, Haifeng Chen