arXiv AI

DCO: Dynamic Cache Orchestration for LLM Accelerators through Predictive Management

The paper proposes DCO, a dynamic cache orchestration scheme for multi-core AI accelerators that uses application-aware policies and dataflow information to guide cache replacement, bypass decisions, and thrashing mitigation. Using a cycle-accurate simulator, the authors demonstrate up to 1.80× speedup over conventional cache architectures and validate the approach with an analytical model and RTL implementation. The design occupies 0.064 mm² on a 15 nm process and operates at 2 GHz, showing that a shared system-level cache can simplify programming while boosting performance for large language model workloads.

arXiv AI

Oct 7

CACHEFORGE: LLM-Guided End-to-End Generative Cache Replacement Policy for Performance and Hardware Efficiency

CACHEFORGE introduces a novel framework that uses a large language model (LLM) to evolve cache‑replacement policies end‑to‑end. In each iteration, the LLM generates new C++ replacement logic, which is evaluated by a trace‑based simulator and refined through reward shaping, structural checks, and mutation. The resulting policies are compact, hardware‑aware, and outperform existing CRC‑2 baselines on SPEC CPU2006, achieving significant improvements in hit rate and IPC across diverse workloads.

By Kaushal Mhapsekar, Bita Aslrousta, Brijesh Kumar Bhayana, Paula Contreras, Azam Ghanbari, Ethan Goodman, Anna Andriiko, Samira Mirbagher Ajorpaz

llms rag

arXiv AI

Jun 9

AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving

arXiv:2606. 09613v1 Announce Type: cross Abstract: Multi-turn LLM agents interleave model calls with external tool invocations, shifting serving from stateless request processing to stateful program execution.

By Rakibul Hasan Rajib, Mengxin Zheng, Qian Lou

llms agents

arXiv Machine Learning

Jun 5

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

arXiv:2606. 06302v1 Announce Type: new Abstract: Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth.

By Hyungmin Kim, Minsoo Kim, Hongseok Kim, Jungwook Choi

llms efficiency

arXiv Machine Learning

Jun 8

Hermes: Accelerating Long-Latency Load Requests via Perceptron-Based Off-Chip Load Prediction

arXiv:2209. 00188v4 Announce Type: replace-cross Abstract: Long-latency load requests continue to limit the performance of high-performance processors.

By Rahul Bera, Konstantinos Kanellopoulos, Shankar Balachandran, David Novo, Ataberk Olgun, Mohammad Sadrosadati, Onur Mutlu

benchmarks

arXiv AI

Jul 7

Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies

arXiv:2411. 07447v5 Announce Type: replace-cross Abstract: LLMs are increasingly used world-wide from daily tasks to agentic systems and data analytics, requiring significant GPU resources.

By Kyoungmin Kim, Jiacheng Li, Kijae Hong, Qunyou Liu, Darong Huang, Anastasia Ailamaki

llms agents

arXiv AI

Aug 10

Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference

arXiv:2604. 26968v2 Announce Type: replace-cross Abstract: Key-value (KV) cache memory management is the primary bottleneck limiting throughput and cost-efficiency in large-scale GPU inference serving.

By Sanjeev Rao Ganjihal

agents efficiency

arXiv Machine Learning

Sep 30

Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging

arXiv:2609.38090v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models are a compelling architecture for scaling model capacity, making them especially attractive for deployment on resource-...

By Sanjali Yadav, Bahar Asgari

efficiency benchmarks

Hugging Face Trending Papers

Jun 8

AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving

Multi-turn LLM agents interleave model calls with external tool invocations, shifting serving from stateless request processing to stateful program execution. Serving these workloads requires scheduling, KV-cache management, and routing policies that use program-level context, including turn dependencies, tool-induced gaps, and reusable KV state.

llms agents

Hugging Face Trending Papers

Jun 4

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth. Non-uniform KV compression effectively preserves more information by considering the individual importance of each KV cache.

llms efficiency

arXiv AI

Aug 11

ArchAgent v2: A Case Study with the Data Prefetching Championship

arXiv:2608. 09874v1 Announce Type: new Abstract: Agentic artificial intelligence has shown great promise in automating algorithm design, but scaling similar techniques to computer microarchitecture discovery remains challenging due to vast search spaces, strict hardware budgets, and long simulation times.

By Abraham Gonzalez, Raghav Gupta, Akanksha Jain, Hanna Alam, Alexander Novikov, Po-Sen Huang, Matej Balog, Marvin Eisenberger, Sergey Shirobokov, Ng\^an V\~u, Hank Levy, Borivoje Nikoli\'c, Sagar Karandikar, Martin Dixon, Parthasarathy Ranganathan

agents

arXiv AI

Jun 30

KernelSight-LM: A Kernel-Level LLM Inference Simulator

arXiv:2606. 28565v1 Announce Type: cross Abstract: As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets.

By Xiteng Yao, Taeho Kim, Hengzhi Pei, Xinle Liu, Kyle Ulrich, Leonard Lausen, Ashish Khetan, Xiang Song, George Karypis, Martin Herbordt

llms efficiency benchmarks

arXiv Computation and Language

Sep 17

Accelerating Stateful Network Applications with Performance Prediction on SoC SmartNICs

arXiv:2410.22229v2 Announce Type: replace-cross Abstract: Offloading stateful network functions to multi-threaded SoC SmartNICs promises significant performance and cost benefits. However, realizing...

By Shaoke Xi, Jiaqi Gao, Fuliang Li, Minlan Yu, Ennan Zhai

efficiency

Related stories