← Back to all news
arXiv Machine Learning By Krishnakumar Balasubramanian

Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attention

Read the original on arXiv Machine Learning →

The paper investigates how unbiased stochastic gradients affect stability in one‑layer linear self‑attention models used for in‑context linear regression. It shows that while full‑batch training preserves an elliptic stability region, mini‑batch sampling introduces batch‑dependent boundaries that drift outward, leading to an exact one‑step crossing criterion and adaptive concentration bounds. The authors also derive drift identities, prove almost‑sure escape in a balanced switching model, and analyze mode leakage and effective context length for softmax attention, corroborated by experiments on causal Transformers.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 1

Switching Linear Attention

Switching Linear Attention (SwiLA) is a new sequence layer that improves upon standard softmax attention by maintaining a fixed-size recurrent state while enhancing representational capacity. It derives its recurrence from a test-time regression framework, using online expectation-maximization in a mixture of linear regressions model. In various benchmarks—including associative recall, in-context language learning, and language modeling—SwiLA achieves strong performance, narrowing the gap to softmax attention and even surpassing it in some settings.

By Hyun Dong Lee, Xavier Gonzalez, Nicolas Zucchet, E. Kelly Buchanan, Emily B. Fox, Scott W. Linderman
efficiencybenchmarks
More like this →
arXiv Machine Learning
Sep 24

Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models

The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.

By Ke Wan, Chen Chen
llmsbenchmarks
More like this →