arXiv AI
Local Inference of LLMs on Long Contexts: A Data Systems Perspective
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Jul 29
InferScale: GPU-Native KV Injection for Personalized LLM Serving
Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.
arXiv Machine Learning
Jul 30
InferScale: GPU-Native KV Injection for Personalized LLM Serving
arXiv:2607. 27090v1 Announce Type: cross Abstract: Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests.
arXiv Machine Learning
Jun 3
Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
arXiv:2606. 02964v1 Announce Type: cross Abstract: Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation.
arXiv Machine Learning
Jun 5
Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving
arXiv:2606. 06302v1 Announce Type: new Abstract: Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth.
Hugging Face Trending Papers
Jun 4
Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving
Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth. Non-uniform KV compression effectively preserves more information by considering the individual importance of each KV cache.
arXiv AI
Jul 7
Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies
arXiv:2411. 07447v5 Announce Type: replace-cross Abstract: LLMs are increasingly used world-wide from daily tasks to agentic systems and data analytics, requiring significant GPU resources.