arXiv Machine Learning
How Language Models Differ in Redistributing Attention-Head Activity Under Serial Demand
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 18
Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Form
arXiv:2608. 15022v1 Announce Type: new Abstract: Language models hold latent quantities in a form they can report on, and more of a quantity is present in that form when the task requires reusing it flexibly.
arXiv AI
Jul 22
A Controlled Study of Attention-Only Transformers
arXiv:2607. 18363v1 Announce Type: cross Abstract: Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once.
arXiv Machine Learning
Jul 27
Learning What Matters: Supervising Sparse Attention Routing with Causal Evidence Sets
arXiv:2607. 21692v1 Announce Type: new Abstract: Sparse attention reduces the cost of long contexts by allowing each query to read only selected parts of the input.
arXiv Machine Learning
Jul 30
Learning What Matters: Supervising Global Context Pruning with Causal Evidence Sets
arXiv:2607. 21692v2 Announce Type: replace Abstract: Sparse attention prunes a long context to the blocks a model needs, and the usual selector is distilled from a dense teacher's attention.
arXiv Machine Learning
Jun 2
Task Structure Reverses Layerwise State Encoding in Sequence Models
arXiv:2606. 00926v1 Announce Type: new Abstract: Mechanistic studies of sequence models often treat layerwise state encodings as architectural traits: recurrent models concentrate readable state, attention-based models distribute it.
arXiv AI
Jul 1
Protein contacts are already in the attention: a single-forward-pass alternative to the Categorical Jacobian
arXiv:2606. 21876v2 Announce Type: replace-cross Abstract: The Categorical Jacobian of Zhang et al.