arXiv Machine Learning October 9, 2026 By Xu Zhao, Jiaming Zhao, Bin Zhao, Yong Yang

Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning

Read the original on arXiv Machine Learning →

The paper introduces QCOC, a method for compressing the key‑value cache used in tabular in‑context learning. QCOC clusters in‑context example states into joint‑KV prototypes, calibrates them against query vectors, and preserves multiplicities to maintain accuracy while achieving high compression. Experiments on 64 OpenML‑CC18 datasets and seven long tables show QCOC outperforms other compression and retrieval methods, achieving up to a 10.5× compression ratio and 508× speed‑up over dynamic retrieval baselines while staying within 0.23 percentage points of full‑context accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 24

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.

By Xiaolin Lin, Jingcun Wang, Olga Kondrateva, Yiyu Shi, Bing Li, Grace Li Zhang

llms efficiency

Hugging Face Trending Papers

Jun 23

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.

llms efficiency

arXiv Machine Learning

Oct 2

PatchKV: Weight-Space Compensation of KV Cache

arXiv:2609.39329v1 Announce Type: new Abstract: Long-context inference with Large Language Models (LLMs) is bottlenecked by the linearly growing memory of the key-value (KV) cache. Existing compressi...

By Chanryeol Lee, Chanhyuk Lee, Yeonwoo Choi, Donggyun Kim, Seunghoon Hong

llms efficiency benchmarks

arXiv AI

Jul 8

FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

arXiv:2607. 06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.

By Anna C\'ordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jes\'us Olivera

llms nlp efficiency benchmarks

arXiv Computation and Language

Sep 23

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

arXiv:2609.25537v1 Announce Type: new Abstract: Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing laten...

By Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia, Yutaka Watanobe, Fang Liu

llms rag efficiency benchmarks

arXiv AI

Sep 30

KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

arXiv:2609.37988v1 Announce Type: new Abstract: As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This i...

By Joao Monteiro, Louis B\'ethune, Anastasiia Filippova, Sonia Laguna, David Grangier, Marco Cuturi

llms efficiency

One email a morning, machine-written

Related stories

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

PatchKV: Weight-Space Compensation of KV Cache

FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

KV-Kaizen: Learning Context-Adaptive Cache Compression Choices