← Back to all news
arXiv Machine Learning By Justin Y. Chen, Ying Feng, Piotr Indyk, Michael Kapralov, Ekaterina Kochetkova, Boris Prokhorov

Attention under Bounded Key-Query Similarity: Space Complexity and Transformer Anisotropy

Read the original on arXiv Machine Learning →

The paper investigates the space complexity of attention mechanisms when key/query cosine similarities are constrained to a small interval around zero, reflecting the anisotropy observed in real transformers. It presents a nearly space‑optimal algorithm alongside a matching lower bound, showing that the distribution of similarities in the lower bound aligns with empirical anisotropy. The study also demonstrates that a simple uniform sampling approach is nearly optimal for typical parameter ranges, explaining its effectiveness on real datasets.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 8

Attention via Black-Box Vector Search

arXiv:2610.10135v1 Announce Type: cross Abstract: Sparse attention mechanisms estimate attention over $n$ tokens using a small subset of keys. Many existing approaches use maximum inner product searc...

By Stepan Zharkov, Krish Singal, Ashwin Padaki, Alexandr Andoni
llmsrag
More like this →
arXiv Machine Learning
Oct 9

Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning

The paper introduces QCOC, a method for compressing the key‑value cache used in tabular in‑context learning. QCOC clusters in‑context example states into joint‑KV prototypes, calibrates them against query vectors, and preserves multiplicities to maintain accuracy while achieving high compression. Experiments on 64 OpenML‑CC18 datasets and seven long tables show QCOC outperforms other compression and retrieval methods, achieving up to a 10.5× compression ratio and 508× speed‑up over dynamic retrieval baselines while staying within 0.23 percentage points of full‑context accuracy.

By Xu Zhao, Jiaming Zhao, Bin Zhao, Yong Yang
llmsefficiency
More like this →
arXiv Machine Learning
Oct 9

Read What Matters: Query-Adaptive Quantization for KV Caches

The paper introduces ReadKV, a query‑adaptive quantization method for KV caches that stores keys and values in progressive codes, allowing each decoding query to allocate different precision prefixes for keys and values. By optimizing a calibrated distortion objective under fixed budgets, ReadKV achieves near‑optimal allocation and demonstrates that query‑dependent access can outperform any query‑independent reader at the same read budget. Experiments on six base models show that reading only four bits on average from an eight‑bit cache increases perplexity by at most 0.66% while reducing logical reads and retained capacity compared to a 16‑bit cache, and it achieves lower latency than existing codecs on an NVIDIA A10G.

By Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian, Shiva Kasiviswanathan
nlpefficiency
More like this →