arXiv Machine Learning
Aug 6
Attention-based representations for multi-task computation
arXiv:2608. 04243v1 Announce Type: new Abstract: Multi-head attention layers produce vector representations that support multiple downstream tasks.
The Head Complexity of Boolean Functions in Single-Layer Attention
arXiv:2609. 04046v1 Announce Type: cross Abstract: What can a single layer of self-attention compute?
Related stories
arXiv Machine Learning
Sep 1
Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers
arXiv:2608.31067v1 Announce Type: new Abstract: Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and lengt...
arXiv Machine Learning
Jun 17
Tight $L_\infty$ Sample Complexity for Low-Degree and Sparse Boolean Polynomials
arXiv:2606. 17319v1 Announce Type: cross Abstract: Motivated by the optimization of bounded binary black-box functions, we study the problem of learning polynomial surrogates over the Boolean hypercube.
arXiv Machine Learning
Jun 11
Higher-Order Token Interactions via Quantum Attention
arXiv:2606. 11673v1 Announce Type: cross Abstract: Standard dot-product self-attention computes, in a single layer, only pairwise (order-2) interactions between tokens; representing a generic order-$k$ interaction is known to require either super-quadratic resources in one layer or composition across depth.
arXiv Machine Learning
Aug 4
Do Neural Networks Really Beat the Curse of Dimensionality? A Bit-Complexity View
arXiv:2608. 01357v1 Announce Type: new Abstract: Traditional approximation theory measures convergence rates in terms of the number of parameters or degrees of freedom.
Hugging Face Trending Papers
Aug 3
Optimal Unambiguous DNFs and Alon-Saks-Seymour
We construct unambiguous DNFs having width $O(n)$ but $0$-certificate complexity $Ω(n^2)$. By utilizing the special structure of these DNFs, we prove a lifting theorem with a constant-sized gadget that lifts the DNF to a communication problem, while losslessly translating the separation in certificate complexity to a separation in communication complexity.
arXiv Machine Learning
Aug 13
Chain-of-Thought Shows the Path to a Tree: Realizing Branching Complexity
arXiv:2608. 11716v1 Announce Type: new Abstract: Chain of Thought (CoT) lifts the expressive ceiling of bounded-depth Transformers, with characterizations tying the number of CoT steps to circuit complexity classes.
arXiv Machine Learning
Jul 27
Indexing: the Beginning and the End
arXiv:2607. 22361v1 Announce Type: new Abstract: We study information bottlenecks in modern deep-learning architectures -- RNNs, softmax transformers, linear-attention transformers and state-space models -- through the lens of the indexing primitive.
arXiv Machine Learning
Aug 4
Optimal Unambiguous DNFs and Alon-Saks-Seymour
arXiv:2608. 02533v1 Announce Type: cross Abstract: We construct unambiguous DNFs having width $O(n)$ but $0$-certificate complexity $\Omega(n^2)$.
arXiv Machine Learning
Aug 5
Provably Learning Multi-Head Attention with Queries
arXiv:2608. 03294v1 Announce Type: new Abstract: We study the problem of learning multi-head softmax attention from black-box input-output access.
arXiv AI
Sep 10
Parity, Sensitivity, and Transformers
arXiv:2602.05896v3 Announce Type: replace-cross Abstract: Understanding what neural architectures can and cannot compute is a central challenge in the theory of AI. One of the fundamental problems in...
arXiv Machine Learning
Aug 13
Three Tokens Force Exponential Feature Rank in Nonnegative Kernel Attention
arXiv:2608. 11427v1 Announce Type: new Abstract: Full attention exposes every token pair, whereas kernel attention compresses a sequence into a fixed-dimensional sketch.