The paper introduces Hierarchical Hash Retrieval (HHR), a coarse‑to‑fine framework designed to improve hash‑based retrieval for large language models. HHR combines Geometry‑Aware Key Routing (GKR) to redistribute feature magnitudes and prune low‑logit keys, with Learned Hash Projection (LHP) to align Hamming distance with true query‑key relevance for fine‑grained retrieval. Experiments on diverse LLMs and benchmarks show that HHR outperforms existing methods, boosting LongBench scores by 1.10 points and achieving up to 3.30× decoding speedup at 128K context length for Llama‑3.1‑8B‑Instruct.
arXiv:2608. 04405v1 Announce Type: cross Abstract: Long-context large language models (LLMs) are increasingly deployed in real-world applications, yet self-attention remains a major efficiency bottleneck -- especially during decoding -- due to the necessity of repeatedly processing ever-growing key-value (KV) caches.
Matryoshka Hash Representations (MHR) propose a two‑stage quantization approach for retrieval‑augmented generation. First, a long binary code is learned; then, frozen, additional zero‑initialized residual adaptors are trained to produce searchable prefixes of varying byte budgets. Evaluated on MS MARCO and transferred to seven BEIR datasets, MHR achieves higher NDCG@10 and Recall@100 at 32‑byte budgets than baselines, especially in low‑budget regimes, and can also improve candidate shortlisting and graph‑index pruning.
Semantic hashing methods for generating short binary hash codes that allow efficient approximate nearest neighbor search in high-dimensional data spaces have gained extensive consideration in recent years. Deep learning-based methods offer better semantic capturing capabilities than traditional approaches relying on manual feature engineering.
Long-context large language models (LLMs) are increasingly deployed in real-world applications, yet self-attention remains a major efficiency bottleneck -- especially during decoding -- due to the necessity of repeatedly processing ever-growing key-value (KV) caches. Existing sparse attention reduce computation by attending to fewer KV pairs, but often suffer from substantial accuracy degradation, require additional training, or rely on expensive hashing.
arXiv:2607. 24567v1 Announce Type: new Abstract: Semantic hashing methods for generating short binary hash codes that allow efficient approximate nearest neighbor search in high-dimensional data spaces have gained extensive consideration in recent years.
arXiv:2510. 04127v2 Announce Type: replace-cross Abstract: Approximate nearest neighbour (ANN) search underpins large-scale retrieval, increasingly within the retrieval-augmented generation pipelines that ground large language models, yet the methods that address it have multiplied across communities until they are seldom read as a single field.
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
arXiv:2608. 15438v1 Announce Type: cross Abstract: Building approximate nearest neighbor (ANN) indexes at billion scale is often dominated by expensive global clustering or graph construction, making time-to-index a first-order systems concern.
arXiv:2605. 16928v2 Announce Type: replace-cross Abstract: Long-context inference in large language models is bottlenecked by the quadratic cost of full attention.
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.
arXiv:2607. 27692v1 Announce Type: cross Abstract: Top-$K$ sparse attention reduces the cost of Softmax and value aggregation by attending to only a small subset of key--value (KV) entries.
arXiv:2609. 31093v1 Announce Type: new Abstract: Scaling language models to long contexts is limited by the quadratic cost of self-attention.