← Back to all news
arXiv Computation and Language By Yuxuan Liu

First Passage Attention

Read the original on arXiv Computation and Language →

The paper introduces First Passage Attention (FPA), a new attention mechanism that replaces the traditional summation of decay contributions with a path‑extrema approach, lowering a key’s logit based on the deepest dip of a learned scalar field over the intervening span. FPA generalises the Forgetting Transformer (FoX) and, when the field and levels share a learned walk, naturally forms a stack of alive keys. Experiments show that a constant‑dimension FPA head can realise any subset of keys left alive by subsequent dips, outperforming softmax heads and FoX baselines in synthetic tasks, and matching FoX perplexity while achieving perfect passkey retrieval at eight times the training length in a 110M‑parameter language model.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Aug 25

Unlearning Is Not Just Erasing: Temporal Decoupling via Generation Inequality

The paper introduces ADU, a fine‑grained training framework that unlearns sensitive information from large language models by decoupling contextual attention pathways instead of erasing tokens. ADU exploits the distinction between local and global attention heads to identify and suppress attention paths that retrieve persistent sensitive anchors, while preserving local‑attention structure and overall language modeling performance. Evaluation on the TOFU and WMDP benchmarks shows ADU achieves superior forget quality (0.93 on TOFU) and retains 92.9% of model utility compared to 81.9% for existing baselines, with fewer side effects in benign contexts.

By Xunlei Chen, Qirui Ye, Yuang Li, Yi Gong, Zhaokun Wang, Wenyi Li, Shiyao Guo, Jinyu Guo
llmsbenchmarkssafety
More like this →
arXiv AI
Jul 7

IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference

arXiv:2605. 25475v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference.

By Xintong Yang, Hao Gu, Binxing Xu, Lujun Li, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Yike Guo, Sirui Han
llmsefficiency
More like this →