The Flow has not summarised this story yet — read it at arXiv Computation and Language.
CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling) is a new method for long-context LLM inference that replaces costly quadratic attention prefilling with a dynamic, input-adaptive sparse routing scheme. It introduces a structural proxy, C_struct, to directly read routing decisions from the proxy attention map, eliminating the need for pooled matrix multiplication and KL divergence. Additionally, CRISP addresses the post-softmax mass cliff by using a sink-aware threshold based on the noise floor, theoretically reducing background noise accumulation to O(n). Empirical results on InfiniteBench, RULER, and LongBench show that CRISP outperforms existing sparse methods and can match or exceed exact dense attention, achieving up to a 5.30× speedup at 512k tokens and significant gains on retrieval-heavy tasks.
arXiv:2609.08189v1 Announce Type: new Abstract: Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens. Existing methods, ho...
arXiv:2606. 04511v1 Announce Type: cross Abstract: Sparse attention reduces compute and memory bandwidth for long-context LLM inference.
arXiv:2608. 19758v1 Announce Type: new Abstract: Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase.
arXiv:2606. 09514v1 Announce Type: new Abstract: Large language models (LLMs) incur high inference cost due to their depth and parameter scale.
arXiv:2606. 06467v1 Announce Type: cross Abstract: Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought.