FastJEV addresses redundancy in JEV models by reusing context and intermediate states during candidate evaluation, eliminating repeated token matching and full backbone execution. It introduces shared context anchoring and candidate prefix sharing to preserve necessary states for branching, and applies decision‑guided pruning based on score changes from a small unlabeled set. Experiments on OmniJev models across five benchmarks show that FastJEV can reduce candidate depth by up to ~45% while maintaining over 94% of original task scores.
Modern language models are built primarily from Transformers, recurrent models, and their hybrid architectures. Transformers rely on token-level attention memories, while recurrent models such as state space models (SSMs) and linear attention maintain compact recurrent states.
arXiv:2608. 02032v1 Announce Type: new Abstract: Modern language models are built primarily from Transformers, recurrent models, and their hybrid architectures.
arXiv:2604. 20920v2 Announce Type: replace Abstract: Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components.
The paper surveys memory‑centric compression techniques for large language models, framing long‑context handling as a memory‑management problem. It distinguishes compute‑active memory from compute‑inactive memory, and models operations such as writing, offloading, recall, and forgetting as state transitions within an information lifecycle. The authors propose a unified cost model and argue that future research should focus on dynamic, lifecycle‑aware memory management rather than static compression.
VisCache introduces a two-stage, plug‑and‑play framework for pruning visual key‑value caches in Vision Large Language Models without retraining. The first stage filters out temporally redundant keyframes, while the second stage, PruneKV, applies a parabolic layer‑wise budget and asymmetric update to selectively prune keys and fuse values, preserving essential context. Experiments show up to 2.35× speedup and significant memory savings with only 19–28% of the original cache retained, outperforming existing baselines.
Raincheck is a framework that defers multimodal prefill for large language models, allowing initial token selection to be decoupled from later information needs. It preserves unselected tokens and progressively prefills them during decoding, using unused compute capacity and scheduling gaps. Experiments on Qwen3-VL-8B and Qwen3.5-9B show that Raincheck recovers a significant portion of quality lost to hard pruning and reduces first-turn time-to-first-token by up to 24.1% with minimal impact on token generation speed.
CatchUp is a training‑free framework that improves efficiency in large vision‑language models by depth‑aware re‑admission of visual tokens. It repeatedly evaluates query‑conditioned relevance, processes only selected tokens per decoder interval, and preserves deferred tokens for later re‑admission, using reference‑to‑current transition signals to approximate missed representation evolution. Experiments on three referring‑expression grounding datasets and six multimodal benchmarks show that CatchUp retains over 92% of full‑model performance with 88.9% fewer active tokens and reduces end‑to‑end latency by 8.0% at a 64‑token budget.
arXiv:2608.30310v1 Announce Type: cross Abstract: Hybrid large language models interleave full-attention layers with linear-attention layers to reduce the cost of long-context inference. This structu...
arXiv:2610.35323v1 Announce Type: cross Abstract: Large language models incur substantial training and inference costs in personalized search and recommendation, as growing interaction histories leng...
arXiv:2609.36835v1 Announce Type: new Abstract: Long-context large language model inference is bottlenecked by KV caches that grow linearly with sequence length. This burden is especially severe for...
arXiv:2608.23843v1 Announce Type: new Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
arXiv:2610.08811v1 Announce Type: new Abstract: As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing me...