arXiv Machine Learning August 4, 2026 By Xingyu Ren, Youran Sun, Chugang Yi, Haizhao Yang

Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation

Read the original on arXiv Machine Learning →

arXiv:2608. 01676v1 Announce Type: cross Abstract: Sparse attention is widely deployed in long-context serving stacks, yet no framework audits how discarding blocks changes the influence of specific content on model output.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 27

Learning What Matters: Supervising Sparse Attention Routing with Causal Evidence Sets

arXiv:2607. 21692v1 Announce Type: new Abstract: Sparse attention reduces the cost of long contexts by allowing each query to read only selected parts of the input.

By Jim Allchin

arXiv Machine Learning

Jun 5

Pattern Selectivity is Not Task-Causal Structure: A Cross-Architecture Mechanistic Study of Composed-Task Circuits in 1B-Class Language Models

arXiv:2606. 05378v1 Announce Type: new Abstract: We test whether a single screen-and-ablate recipe -- identify attention-head circuits by task-pattern selectivity, then verify by causal ablation against a matched-random null -- produces consistent mechanistic claims across model families.

By Yongzhong Xu

llms benchmarks

arXiv AI

Aug 5

Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks

arXiv:2608. 03297v1 Announce Type: new Abstract: A standard claim in the literature on retrieval-augmented and memory-augmented language models is that shorter context is better when the relevant information is preserved.

By Mohsen Arjmandi

llms rag benchmarks

arXiv Machine Learning

Aug 6

When More Becomes Less: Position-Dependent Repetition Effects in Language Models

arXiv:2608. 04021v1 Announce Type: cross Abstract: Cloze-style probes that vary how often a target token appears implicitly assume that more copies of a target affect prediction the same way regardless of where the readout slot sits.

By Han-yu Wang

llms benchmarks

arXiv Machine Learning

Jul 30

Learning What Matters: Supervising Global Context Pruning with Causal Evidence Sets

arXiv:2607. 21692v2 Announce Type: replace Abstract: Sparse attention prunes a long context to the blocks a model needs, and the usual selector is distilled from a dense teacher's attention.

By Jim Allchin

efficiency

arXiv AI

Jul 16

The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicators

arXiv:2607. 13075v1 Announce Type: cross Abstract: Context can change whether a request is harmful without changing its topic or surface form.

By Dominik Schwarz

Related stories

Learning What Matters: Supervising Sparse Attention Routing with Causal Evidence Sets

Pattern Selectivity is Not Task-Causal Structure: A Cross-Architecture Mechanistic Study of Composed-Task Circuits in 1B-Class Language Models

Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks

When More Becomes Less: Position-Dependent Repetition Effects in Language Models

Learning What Matters: Supervising Global Context Pruning with Causal Evidence Sets

The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicators