The paper introduces SympFormer, a framework that derives accelerated attention blocks from inertial, Nesterov-type damped Hamiltonian dynamics on spaces of probability densities. It develops new geometries—local-mobility for Otto-Wasserstein and softmax-Wasserstein, and a quotient Stein geometry—to rigorously formulate these dynamics, providing gradient-flow and Hamiltonian foundations for softmax and linear attention. The authors analyze existence conditions, link continuous moment blow-up to discrete layer depth, and propose geometric corrections that empirically improve next-token prediction in decoder-only models.
arXiv:2605. 10931v2 Announce Type: replace-cross Abstract: Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models.
The paper introduces a new optimal transport framework that uses the Barron energy—an intrinsic neural network complexity measure—instead of the traditional kinetic energy. By modeling transport as deep residual networks that map a reference distribution to a target distribution, the authors define a new metric on probability measures and analyze its properties for generative modeling. They demonstrate that diffusion generative models are suboptimal in this Barron geometry and explore how adaptivity can improve alternative generative approaches.
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
arXiv:2609.25659v1 Announce Type: new Abstract: Many scientific datasets, such as molecular conformational ensembles or single-cell tissue measurements, are naturally modeled as meta-distributions: d...
arXiv:2604. 09560v2 Announce Type: replace Abstract: Softmax attention is the row-normalized operator of a diffusion map: both normalize a learned score into a Markov operator, and differ only in what the score is allowed to contain.
RW-Flow presents a new one‑step generative framework for data on compact manifolds, leveraging Wasserstein gradient flows. The authors derive a necessary and sufficient identifiability condition for velocity fields on compact, connected Riemannian manifolds, showing that a symmetric, Lipschitz‑continuous cost function yields identifiability iff its Gibbs kernel is nondegenerate. Experiments on geospatial events, protein and RNA torsion angles, and discretized manifolds demonstrate that RW‑Flow surpasses existing one‑step methods across most benchmark settings.
arXiv:2602. 18849v2 Announce Type: replace-cross Abstract: We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation.
arXiv:2608. 01434v1 Announce Type: new Abstract: Normally the statistical mechanics of learning treats constraints on weight distributions as restrictions that shrink the space of possible solutions.
arXiv:2605. 26078v3 Announce Type: replace Abstract: Wasserstein policy gradient (WPG) is a policy optimization method for reinforcement learning (RL) that exploits the optimal-transport geometry of action distributions.
arXiv:2505. 06589v2 Announce Type: replace-cross Abstract: Modern machine learning repeatedly manipulates probability measures: empirical datasets, generated samples, latent distributions, class-conditional laws, particle systems, weights of wide networks and attention patterns.
arXiv:2511. 05940v3 Announce Type: replace-cross Abstract: Score-based diffusion models have emerged as a powerful class of generative methods, with successful applications across diverse domains.
arXiv:2608. 11544v1 Announce Type: cross Abstract: We propose CVaR-penalized Generative Particle Algorithm (CVaR-GPA), a robust, tail-agnostic algorithm for fine-tuning generative models to learn heavy-tailed distributions and capture extreme events, requiring no prior knowledge or estimation of the target's tail characteristics.