BayesNDE is a neural density estimator that uses Bayesian generative modeling to estimate densities without relying on invertible networks or Jacobian-determinant calculations. It constructs an adaptive proposal for each observation by inferring a sample-specific latent posterior, and then applies bridge sampling to combine proposal samples with separate posterior samples for density estimation. Experiments on synthetic datasets show improved density estimation and structure recovery, while real-world applications demonstrate better anomaly detection.
By Chenglin Li, Qiao Liu
arXiv:2606. 29256v1 Announce Type: cross Abstract: In recent years, models based on the Transformer architecture have seen widespread applications and have become one of the core tools in the field of deep learning.
By Peilin Liu, Ding-Xuan Zhou
The paper investigates diffusion models trained in a lazy high‑dimensional regime, extending benign overfitting theory to generative settings. By analyzing denoising score matching in a vector‑valued RKHS with an inner‑product kernel, the authors derive exact risk trajectories under gradient flow when the number of samples scales proportionally with dimensionality. These trajectories reveal three distinct phases—spectral generalization, noise‑dominated interpolation, and empirical Bayes memorization—whose interplay shapes the distribution of generated samples.
By Hugo Latourelle-Vigeant, Sinho Chewi, Aram-Alexandre Pooladian, John Sous, Theodor Misiakiewicz
arXiv:2605. 13092v2 Announce Type: replace-cross Abstract: Density estimation in high-dimensional settings is an important and challenging statistical problem.
By Ruitong Zhang, Ke Deng
The paper introduces Cubit, a Transformer‑style architecture that replaces the standard attention mechanism with Kernel Ridge Regression (KRR). By interpreting attention as Nadaraya‑Watson regression, Cubit incorporates the closed‑form KRR solution, combining kernel‑based value aggregation with normalization via the inverse kernel matrix. The authors also propose a Limited‑Range Rescale (LRR) to stabilize training and report that Cubit shows improved long‑sequence modeling, with gains increasing as training sequence length grows.
By Chuanyang Zheng, Jiankai Sun, Yihang Gao, Yuehao Wang, Liangchen Tan, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, Xiaodong Liu
The paper introduces attention kernels that replace the exponential function in transformer softmax to better handle operator learning on probability measures with heavy-tailed (polynomial) distributions. Two new benchmarks with closed‑form targets are constructed to evaluate how different kernel growth rates and data preprocessing affect performance. The study finds that slower‑growing kernels prevent ensemble collapse on heavy‑tailed tasks, while softmax with symlog preprocessing only succeeds on a subset of problems, and that all kernels perform similarly on Gaussian data.
By Kailen Hargenrader, Edoardo Calvello, Bohan Chen
arXiv:2607. 23869v1 Announce Type: cross Abstract: Randomized features provide a scalable approximation to kernel machines, but their performance depends strongly on the choice of feature distribution.
By Masoud Badiei Khuzani, Sharath Honnaiah, Atiq Islam, Alex Cozzi, Abraham Bagherjeiran
arXiv:2606. 01172v1 Announce Type: new Abstract: Modeling unknown latent functions from finite, irregularly sampled measurements is a recurring challenge across science and engineering.
By Peiman Mohseni, Nick Duffield, Raymond K. W. Wong
The paper introduces an amortized learning framework for selecting bandwidths in kernel density estimation by optimizing the logarithmic score across a distribution of tasks. It uses a truncated-and-renormalized bounded-support formulation and affine standardization to achieve stable learning and transferability across different intervals. Experiments on Gaussian samples, a multi-family benchmark, and randomized Gaussian mixtures demonstrate that the learned selector outperforms traditional methods such as Silverman’s rule, Sheather–Jones, and least‑squares cross‑validation, especially for small or heterogeneous samples.
By Junyi Liang, Hailiang Du
Sufficiently Reduced Distributional Regression (SRDR) is a generative approach that merges conditional distribution estimation with nonlinear sufficient dimension reduction (SDR). By framing SDR as a risk minimization problem using strictly proper scoring rules, SRDR jointly learns a dimension reduction map and a generative prediction model through minimization of the energy score, which can be estimated via sampling. The method extends to multi‑environment data and classification, and theoretical results show convergence of estimated conditional distributions in energy distance, implying asymptotic sufficiency. In experiments on CT slice localization, superconductivity data, and digit classification, SRDR recovers low‑dimensional sufficient structure and matches or surpasses state‑of‑the‑art nonlinear SDR methods in representation quality and predictive performance.
By Alexander Henzi, Tiange Liu, Xinwei Shen
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.
By Val\'erie Castin, Pierre Ablin, Jos\'e Antonio Carrillo, Gabriel Peyr\'e
arXiv:2502. 04646v2 Announce Type: replace-cross Abstract: Weighted sampling -- sampling from a probability density function (PDF) proportional to the product of a base PDF and a weight function -- is a fundamental technique with wide-ranging applications in variance reduction, biased sampling, data augmentation, and more.
By Heasung Kim, Taekyun Lee, Hyeji Kim, Gustavo de Veciana