The paper introduces SAKI-RoPE, a training‑free low‑rank key‑indexing method that optimally preserves attention scores by solving a covariance‑weighted rank‑r problem using query and key statistics. It outperforms key‑PCA across multiple large language models and context lengths, achieving higher recall with significantly lower rank. The authors also analyze random‑matrix noise to justify the calibration token budget and demonstrate that clipping the Marchenko‑Pastur bulk improves performance when calibration data is limited.
arXiv:2608. 03228v2 Announce Type: replace Abstract: Existing low rank KV cache methods preserve either model weights or key variance, neither of which directly reflects the attention scores used during inference.
arXiv:2608. 03228v1 Announce Type: new Abstract: Existing low rank KV cache methods preserve either model weights or key variance, neither of which directly reflects the attention scores used during inference.
arXiv:2410.02343v2 Announce Type: replace Abstract: Large language models (LLMs) routinely fail to output the correct option in multiple-choice question answering (MCQA) while encoding the answer int...
arXiv:2609.30738v1 Announce Type: new Abstract: KV cache eviction methods such as SnapKV and PyramidKV rank tokens solely by mean attention over a small observation window. We study a unified score,...
arXiv:2607. 15498v1 Announce Type: cross Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference.
arXiv:2609.00762v1 Announce Type: new Abstract: Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where...
The paper investigates whether the rank of latent matrices in matrix‑chain‑of‑thought (Matrix‑CODI) models influences performance on reasoning tasks. Across multiple training regimes on ProsQA and GSM8K‑Aug, rank‑k projection ablations show flat accuracy curves, indicating that truncating the latent matrix to low rank does not hurt performance. Experiments with various readout architectures—bilinear, bilinear‑plus‑GELU, SVD‑augmented, and quadratic—confirm that rank‑indifference persists even for nonlinear readouts, and a linear probe on the latent matrix underperforms a raw pretrained hidden state.
arXiv:2606. 07596v1 Announce Type: new Abstract: Fine-tuning often introduces spurious correlations alongside task knowledge, causing systematic failures on underrepresented groups.
arXiv:2609.38901v1 Announce Type: new Abstract: Representer explanations rank the training landmarks that most influence a self-supervised representation. At scale, this ranking rests on up to four s...
arXiv:2608. 06849v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by quadratic attention computation and growing KV-cache costs.
The paper introduces Stratified Sparse Attention (SSA), a framework that blends deterministic top‑k retrieval with score‑guided sampling to reduce the gap between sparse and dense attention in large language models. SSA attends to high‑scoring tokens deterministically while sampling the remaining attention mass using proxy scores, thereby recovering more of the attention distribution. Experiments on benchmarks such as RULER, LongBench, LOFT, and AIME show that SSA outperforms existing sparse methods across models ranging from 3B to 27B parameters, achieving significant accuracy gains and up to 2.96× decode speedup with optimized CUDA kernels.
Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where those coefficients act is equally consequential....