The paper investigates how the geometry of token embeddings influences learning in a minimal token‑perceptron model, where a linear predictor maps sequences of embedded tokens to outputs. Using the replica method, it derives generalization errors for continuous spherical embeddings and for discrete vocabulary embeddings, revealing that learning thresholds depend on embedding dimension and that optimal learners prioritize directions of greatest variance in the embedding space. The study also quantifies storage capacity and recovery thresholds for noiseless regression and random associations.
arXiv:2606. 02765v1 Announce Type: cross Abstract: Model dimension ($d_{model}$) is a fundamental hyperparameter in transformer language models, yet its role in setting the geometric limits of feature representation remains under-explored.
arXiv:2609.38834v1 Announce Type: cross Abstract: Contrastive learning is a successful paradigm for learning $d$-dimensional geometric representations from a collection of ``anchor--positive--negativ...
arXiv:2609. 02155v1 Announce Type: new Abstract: The Johnson-Lindenstrauss (JL) lemma guarantees that a random projection of $n$ points to $m=O(\varepsilon^{-2}\log n)$ dimensions preserves pairwise squared distances within relative error $\varepsilon$ with high probability, and this dimension order is asymptotically optimal.
arXiv:2608. 15472v1 Announce Type: cross Abstract: The problem of networked information aggregation, studied in Kearns et al.
arXiv:2004. 05813v3 Announce Type: replace-cross Abstract: Suppose that we are given independent, identically distributed random samples $x_1,\cdots,x_n$ from a mixture at most $k$ many $d$-dimensional spherical Gaussian distributions $\mu_1,\cdots,\mu_{k_0}$ of identical and known variance $\sigma^2$ in each coordinate, such that the minimum $\ell^2$ distance between two distinct centers $y_l$ and $y_j$ is greater than $2\Delta\sigma \min\{\sqrt{d},\sqrt k\}$, where $\Delta>C_0$, and $C_0$ is a sufficiently large universal constant.
arXiv:2608. 06337v1 Announce Type: cross Abstract: A monotone adversary observes an i.
arXiv:2605. 05189v2 Announce Type: replace-cross Abstract: How many key-value associations can a $d\times d$ linear memory store?
arXiv:2610.27402v1 Announce Type: cross Abstract: Motivated by the notion of features from machine learning, we study the complexity of representing data that allows for evaluation with a given famil...
arXiv:2606. 28573v1 Announce Type: new Abstract: Modern machine learning models are trained by optimizing high-dimensional non-convex empirical risk functions.
arXiv:2608. 15215v1 Announce Type: cross Abstract: Token-level knowledge distillation (KD) matches two conditional distributions per position, yet the standard objectives compare them pointwise: a Kullback-Leibler gradient is blind to which wrong token receives probability mass.
arXiv:2609.15825v1 Announce Type: new Abstract: A self-supervised encoder is trained once, frozen, and reused through lightweight probes on tasks nobody named at training time; the practitioner's que...
arXiv:2407. 00966v3 Announce Type: replace Abstract: In traditional models of supervised learning, the goal of a learner-- given examples from an arbitrary joint distribution on $\mathbb{R}^d \times \{\pm 1\}$-- is to output a hypothesis that is competitive (to within $\epsilon$) of the best fitting concept from some class.