arXiv Machine Learning
Conditioned Initialization for Attention
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 30
Generalization Analysis of Transformers in Distribution Regression
arXiv:2606. 29256v1 Announce Type: cross Abstract: In recent years, models based on the Transformer architecture have seen widespread applications and have become one of the core tools in the field of deep learning.
arXiv Machine Learning
Jun 16
Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
arXiv:2606. 16429v1 Announce Type: new Abstract: Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retaining much of the quality of Transformer models.
arXiv Machine Learning
Aug 11
Training-Free Universal Approximation by Prompting Random Transformers
arXiv:2608. 09558v1 Announce Type: new Abstract: How expressive is prompting a transformer?
arXiv Machine Learning
Sep 30
Procedural Core: A Compact Recurrent Initialization for Vision Transformers
arXiv:2609.37631v1 Announce Type: new Abstract: Transformers are typically trained from random initialization, requiring all their capabilities to emerge from large-scale optimization. Recent work sh...
arXiv Machine Learning
Oct 21
Transformers Learn Two Competing Sorting Algorithms, and Only One Length-Generalizes
arXiv:2610.26974v1 Announce Type: new Abstract: We show that two-layer transformers trained to near-zero loss on a sorting task converge to one of two mechanistically distinct solution classes depend...
arXiv Machine Learning
Jul 16
Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Models
arXiv:2607. 13395v1 Announce Type: new Abstract: The pursuit of autonomously self-improving models has attracted growing interest in the era of large-scale foundation models.