arXiv:2610. 34553v1 Announce Type: new Abstract: How efficiently can transformers store factual knowledge?
arXiv:2610.07676v1 Announce Type: cross Abstract: Research on transformer expressivity shows whether a transformer is capable of solving a given task, but gives little indication of whether the solut...
The paper presents quantitative generalization bounds for fixed‑precision transformer models, addressing both in‑distribution statistical generalization and length generalization. It derives guarantees based on model parameters, quantization space, and alphabet size, and shows how training length and sample size trade off, with longer training being exponentially more efficient. Experiments confirm the theoretical predictions.
arXiv:2601. 22002v5 Announce Type: replace Abstract: Transformers achieve superior performance on many tasks, but impose heavy compute and memory requirements during inference.
arXiv:2607. 10034v1 Announce Type: new Abstract: Large language models (LLMs) store factual knowledge in their parameters.
The paper examines how to allocate attention heads and head dimensions across Transformer layers to balance expressivity and efficiency. It provides a mathematical analysis of early layers’ role in information extraction and characterizes the trade‑off between head count and dimension under a fixed parameter budget. The authors prove a saturation effect of softmax activations, showing that increasing head dimensions yields diminishing returns, especially for long sequences, and propose strategies for efficient parameter allocation across layers.
arXiv:2610.32104v1 Announce Type: cross Abstract: Several formal theories have been developed to characterize the capabilities and limitations of the transformer architecture, each employing differen...
arXiv:2604.13082v3 Announce Type: replace-cross Abstract: Large language models today still make mistakes on multi-digit arithmetic when they do not have tools, and part of the reason is how numbers...
arXiv:2609.08851v1 Announce Type: new Abstract: Recent advancements in transformer length generalization theory enable us to reliably predict when a transformer can learn to solve a task. In particul...
arXiv:2512. 22088v3 Announce Type: replace-cross Abstract: The scaling law, a cornerstone of Large Language Model (LLM) development, predicts improvements in model performance with increasing computational resources.
arXiv:2511. 12081v2 Announce Type: replace-cross Abstract: Despite massive investments in scale, deep models for click-through rate (CTR) prediction often exhibit rapidly diminishing returns -- a stark contrast to the {predictable scaling laws} seen in large language models (LLMs).
arXiv:2605. 22223v2 Announce Type: replace Abstract: We study how we can leverage only a handful of characteristics of a transformer's architecture to closely predict the number of different sequences it can output, both qualitatively and quantitatively.
arXiv:2607. 17624v1 Announce Type: new Abstract: Transformers are remarkably versatile and their design is largely consistent across a variety of applications.