arXiv Statistics ML

Generative sequence modeling for infinite memory processes via predictive states

arXiv Statistics ML
Oct 26

Empirical concentration for chains with infinite memory

The paper presents empirical concentration inequalities for stationary chains with infinite memory, extending previous results that were limited to Markov chains. It establishes a bounded‑differences inequality under a summable Wasserstein contraction condition and shows how to estimate the necessary dependence coefficients from a single trajectory, using transition counts for finite state spaces or kernel density derivatives for continuous spaces. The authors provide general error bounds that reflect the memory tail and demonstrate that, after applying a fixed safety factor, the resulting empirical bounds are asymptotically as tight as the theoretical ones, while remaining fully adaptive without requiring knowledge of the underlying constants.

By Pierre Alquier, Paul Doukhan
arXiv Machine Learning
Jul 28

Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions

arXiv:2506. 05678v3 Announce Type: replace Abstract: The evolution of sequence modeling architectures, from recurrent neural networks and convolutional models to Transformers and structured state-space models, reflects ongoing efforts to address the diverse temporal dependencies inherent in sequential data.

By Haotian Jiang, Zeyu Bao, Shida Wang, Qianxiao Li
arXiv AI
Aug 7

The Impossibility Triangle of Long-Context Modeling

arXiv:2605. 05066v2 Announce Type: replace-cross Abstract: We identify and prove a fundamental trade-off governing long-sequence models: no model can simultaneously achieve (i) per-step computation independent of sequence length (Efficiency), (ii) state size independent of sequence length (Compactness), and (iii) the ability to recall a number of historical facts proportional to sequence length (Recall).

By Yan Zhou
arXiv Statistics ML
Sep 21

Neural composite likelihood estimation: simulation based inference for time series

Neural Composite Likelihood Estimation (NCLE) extends simulation‑based inference to high‑dimensional time series by partitioning long sequences into equal‑sized batches. For each batch, a neural network estimates the likelihood via conditional density estimation, and the product of these batch likelihoods forms an approximate composite likelihood. Frequentist inference is then performed by maximizing this composite likelihood to obtain a point estimate and by estimating the Godambe information matrix to derive confidence intervals.

By Grace Yan, Mark Beaumont, Dennis Prangle
Hugging Face Trending Papers
Sep 17

Next-token functional estimation

Suppose we observe the first $n$ points of a sequence of random variables having length $n+1$, and wish to estimate a functional of the unobserved final point and the empirical measure of the $n$ obse...

arXiv Machine Learning
Sep 11

Particle GFlowNets: Rethinking Generative Marginalization Models

The paper introduces Particle GFlowNets, showing that Generative Marginalization Models (MaMs) are equivalent to Generative Flow Networks. It extends MaMs to non‑autoregressive sampling and proposes an automatic full‑state rejuvenation criterion based on the Gelman‑Rubin statistic to accelerate learning. Experiments demonstrate significant training speedups in large combinatorial spaces.

By Tiago da Silva, Diego Mesquita, Salem Lahlou