arXiv AI August 25, 2026 By Jingquan Chen, Jie Feng, Jinghua Piao, Shaogang Hu, Yong Li

CacheSpec: Finding the Sweet Spot for Small Models in Large Language Models

CacheSpec is an inference optimization framework that transforms Program-of-Thoughts (PoT) style programs into reusable cache objects for large language models. By employing a small model for semantic variable extraction on cache hits and speculative drafting during target-LLM generation, CacheSpec reduces inference latency and improves cache reuse. Experiments on shopping, web, formula, and code QA datasets demonstrate up to 3.1× speedup in latency and 2.8× throughput gains over traditional PoT methods, while maintaining or improving task quality.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 24

MiniCache: Reusable Program Caching with Small Model Interfaces for Efficient LLM Inference

arXiv:2607. 20507v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for program-aided reasoning, agentic decision making, and structured task execution, but these applications often incur high inference cost.

By Jingquan Chen, Jinghua Piao, Jie Feng, Shaogang Hu, Yong Li

llms agents efficiency

arXiv Machine Learning

Sep 10

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

arXiv:2609.07966v1 Announce Type: new Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-cont...

By Michael Wang, Keith Li, Roozbeh Bostandoost

llms efficiency

arXiv Computation and Language

Oct 8

SEER: Self-Enhancing Chain-of-Thought Compression for Reasoning Models

arXiv:2509.14093v3 Announce Type: replace-cross Abstract: Chain-of-Thought (CoT) prompting can substantially improve the reasoning ability of large language models (LLMs), but it often comes with hig...

By Kerui Huang, Shuhan Liu, Xing Hu, Tongtong Xu, Lingfeng Bao, Xin Xia

llms benchmarks

arXiv AI

Aug 6

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

arXiv:2608. 04771v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost.

By Qiyuan Zhu, Dezhi Li, Pengyu Cheng, Tianle Chen, Jiacheng Wang, Ruijie Shen, Hao Gu, Sida Lin, Zirui Liu, Jiacheng Liu, Sirui Han

benchmarks

Hugging Face Trending Papers

Aug 5

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost. KV-cache compression is a common solution, yet existing reasoning-oriented methods apply a uniform policy across the trajectory and judge compression only by what it removes from the cache.

benchmarks

arXiv Computation and Language

Aug 24

Self-Speculation for Faster Reasoning Models

arXiv:2608.20359v1 Announce Type: new Abstract: Large language models (LLMs) are deployed for increasingly complex tasks involving planning and multi-step decision making, but high-quality performanc...

By Ravisri Valluri, Tung Nguyen, Aditya Grover

llms agents

One email a morning, machine-written

Related stories

MiniCache: Reusable Program Caching with Small Model Interfaces for Efficient LLM Inference

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

SEER: Self-Enhancing Chain-of-Thought Compression for Reasoning Models

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

Self-Speculation for Faster Reasoning Models