arXiv AI July 24, 2026 By Jingquan Chen, Jinghua Piao, Jie Feng, Shaogang Hu, Yong Li

MiniCache: Reusable Program Caching with Small Model Interfaces for Efficient LLM Inference

arXiv:2607. 20507v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for program-aided reasoning, agentic decision making, and structured task execution, but these applications often incur high inference cost.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Aug 25

CacheSpec: Finding the Sweet Spot for Small Models in Large Language Models

CacheSpec is an inference optimization framework that transforms Program-of-Thoughts (PoT) style programs into reusable cache objects for large language models. By employing a small model for semantic variable extraction on cache hits and speculative drafting during target-LLM generation, CacheSpec reduces inference latency and improves cache reuse. Experiments on shopping, web, formula, and code QA datasets demonstrate up to 3.1× speedup in latency and 2.8× throughput gains over traditional PoT methods, while maintaining or improving task quality.

By Jingquan Chen, Jie Feng, Jinghua Piao, Shaogang Hu, Yong Li

llms agents efficiency

arXiv AI

Sep 16

Comparative Characterization of KV Cache Management Strategies for LLM Inference

arXiv:2604.05012v2 Announce Type: replace-cross Abstract: Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value...

By Oteo Mamo, Olga Kogiou, Hyunjin Yi, Weikuan Yu

llms efficiency benchmarks

arXiv AI

Aug 18

From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems

arXiv:2608. 15127v1 Announce Type: cross Abstract: Agentic applications are shifting AI serving from isolated model inference to long-running workloads in which LLMs coordinate tools, environments, and persistent state.

By Chaokun Chang, Yukun Zhou, Kaihua Fu, Dakai An, Tianyu Feng, Hanfeng Lu, Sheng Yao, Pu Guo, Yinghao Yu, Yizhou Shan, Bo Li, Binhang Yuan, Wei Wang

llms agents benchmarks

arXiv Machine Learning

Jun 19

Closing the Calibration Gap in Semantic Caching

arXiv:2606. 19719v1 Announce Type: cross Abstract: Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries.

By Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev, Srijith Rajamohan, Jen Agarwal

llms

arXiv Machine Learning

Sep 10

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

arXiv:2609.07966v1 Announce Type: new Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-cont...

By Michael Wang, Keith Li, Roozbeh Bostandoost

llms efficiency

arXiv Computation and Language

Aug 21

ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents

arXiv:2608. 19662v1 Announce Type: new Abstract: Agentic language models repeatedly encode tool and skill schemas that recur across requests in different combinations and orders, preventing standard prefix caching from reusing their key--value (KV) states.

By Yichu Fang, Sitong Wei, Haozhe Hu, Xiaoyu Shen

llms agents nlp efficiency benchmarks

One email a morning, machine-written

Related stories

CacheSpec: Finding the Sweet Spot for Small Models in Large Language Models

Comparative Characterization of KV Cache Management Strategies for LLM Inference

From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems

Closing the Calibration Gap in Semantic Caching

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents