← Back to all news
arXiv Machine Learning By Saipraveen Vabbilisetty, Ajay Kumar Boddepalli, Deep Narayan Mishra, Shashank Kapadia, Haoan Wang, Anupriya Sharma

Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs

Read the original on arXiv Machine Learning →

The paper introduces the Tri‑Metric Router, a deterministic, training‑free policy that chooses among Raw, Neural, and Lexical pipelines for retrieval‑augmented generation on commodity GPUs. It uses three CPU‑side signals—spatial complexity, syntactic density, and type‑token ratio—to balance VRAM headroom and latency, calibrated on LongBench qasper. The method eliminates out‑of‑memory failures and improves alignment and F1 scores compared to always‑on lexical compression without extra VRAM or training costs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 21

From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG

arXiv:2608. 19535v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) improves language-model responses by grounding generation in external passages, which comes with overhead: retrieved context lengthens the prompt, increasing prefill work, KV-cache footprint, memory traffic, latency, and energy.

By Zlatan Feric, Amir Taherin, Yanzhi Wang, David Kaeli
llmsragefficiencybenchmarks
More like this →
arXiv AI
Jun 24

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.

By Xiaolin Lin, Jingcun Wang, Olga Kondrateva, Yiyu Shi, Bing Li, Grace Li Zhang
llmsefficiency
More like this →
arXiv Computation and Language
Oct 16

From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models

The paper surveys memory‑centric compression techniques for large language models, framing long‑context handling as a memory‑management problem. It distinguishes compute‑active memory from compute‑inactive memory, and models operations such as writing, offloading, recall, and forgetting as state transitions within an information lifecycle. The authors propose a unified cost model and argue that future research should focus on dynamic, lifecycle‑aware memory management rather than static compression.

By Chaogui Gou
llmsagentsbenchmarks
More like this →