The paper examines whether context compression, which encodes inputs into compact representations while preserving semantic content, maintains refusal alignment in instruction-tuned language models. It finds that encoder-based compression consistently weakens refusal behavior across various model families and architectures, even though reconstruction fidelity remains high. Mechanistic analysis attributes this to attenuation of activation along the decoder’s learned refusal direction, and introduces Memory Steering—a lightweight inference-time intervention that restores refusal rates without retraining.
Context compression reduces inference cost by encoding inputs into compact representations while preserving semantic content. An open question is whether semantic preservation alone is sufficient to m...
The paper reports a Size‑Fidelity Paradox in large‑language‑model (LLM) context compression: larger compressors can produce lower reconstruction error yet yield less faithful reconstructed contexts. Two key mechanisms drive this effect: knowledge overwriting, where larger models replace source facts with their own beliefs, and semantic drift, where they paraphrase or restructure content. Across 27 compressor setups, mid‑sized models often recover contexts more faithfully, and analysis of embedding geometry shows that larger models organize memory into broader semantic subspaces, leading to ambiguous representations that weaken recovery.
arXiv:2609.27510v1 Announce Type: cross Abstract: Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and...
arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.
arXiv:2609.25537v1 Announce Type: new Abstract: Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing laten...
Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and undermining the reliability of evaluation results....
Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt.
arXiv:2608.00311v2 Announce Type: replace Abstract: Long-context inference with large language models (LLMs) is costly: self-attention during prefill scales quadratically with sequence length, and th...
arXiv:2610.10304v1 Announce Type: new Abstract: We study whether latent sequence compression of prompt prefixes preserves the capabilities that large language models rely on during inference. We intr...
arXiv:2608. 11249v1 Announce Type: cross Abstract: We study the problem of lossless text compression, motivated by the rapid growth in the collection and storage of digital textual data - including plain text, source code, and structured formats such as XML - and by recent advances in neural language model-based compression.
The paper introduces Variance‑Calibrated Modulation (VCM), a training‑free pre‑decoding technique that reshapes language model probability distributions before truncation. VCM uses two dynamic mechanisms: a Contextual Searchlight via PMI to suppress stopwords and highlight context‑relevant tokens, and an Adaptive Self‑Debiasing that applies scale‑invariant penalization based on real‑time logit standard deviation. Experiments on open‑ended generation, factual QA, and mathematical reasoning show that VCM consistently reduces the likelihood trap, improving diversity, coherence, and reasoning accuracy with minimal computational cost.
arXiv:2509. 14230v2 Announce Type: replace Abstract: While structured pruning presents a highly effective pathway for accelerating Large Language Model (LLM) inference, existing methods frequently suffer from significant performance degradation and demand computationally retraining to recover capabilities.