arXiv Machine Learning
Jun 2
The Entropic Signature of Class Speciation in Diffusion Models
arXiv:2602. 09651v2 Announce Type: replace-cross Abstract: Diffusion models do not recover semantic structure uniformly over time.
Breakdown of Local Denoising as Semantic Speciation
Related stories
arXiv Machine Learning
Jul 10
An exact information theory of generalization phase transitions in Bayesian diffusion models
arXiv:2607. 08041v1 Announce Type: new Abstract: How diffusion models circumvent the curse of dimensionality to learn complex distributions over high dimensional spaces from a finite training set, instead of memorizing it, remains a fundamental mystery.
Hugging Face Trending Papers
Jul 9
An exact information theory of generalization phase transitions in Bayesian diffusion models
How diffusion models circumvent the curse of dimensionality to learn complex distributions over high dimensional spaces from a finite training set, instead of memorizing it, remains a fundamental mystery. To address this, we introduce analytically tractable Bayesian information restricted diffusion (BIRD) models, in which each pixel observes restricted information about noisy data.
Hugging Face Trending Papers
Jul 27
Context Is King: How In-Context Specification Shapes the Geometry of Concepts
Large language models place structured concepts on geometrically faithful manifolds: weekdays lie on a circle, months on another, usually taken to be a fixed world-model the network stores and looks up. We show that context is king: the structure a model actually uses is set by the in-context specification.
arXiv Machine Learning
Jul 28
Context Is King: How In-Context Specification Shapes the Geometry of Concepts
arXiv:2607. 24425v1 Announce Type: new Abstract: Large language models place structured concepts on geometrically faithful manifolds: weekdays lie on a circle, months on another, usually taken to be a fixed world-model the network stores and looks up.
arXiv AI
Jul 8
Few Channels Draw The Whole Picture: Revealing Massive Activations in Diffusion Transformers
arXiv:2605. 13974v2 Announce Type: replace-cross Abstract: Diffusion Transformers (DiTs) and related flow-based architectures are now among the strongest text-to-image generators, yet the internal mechanisms through which prompts shape image semantics remain poorly understood.
arXiv Machine Learning
Jun 16
Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design
arXiv:2606. 15327v1 Announce Type: new Abstract: Diffusion Language Models (DLMs) have demonstrated strong scaling capacity as alternatives to autoregressive language models.
arXiv Machine Learning
Sep 4
Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models
The paper introduces a new measure of generative‑process diversity for language models, using Normalised Compression Distance on raw outputs after controlling for permutation effects. Across 38 models, this metric uncovers population structure that semantic similarity misses and predicts lower correlated failures across ten benchmark families, independent of semantic similarity or model capability. The authors argue that higher generative‑process diversity reduces correlated failures in multi‑model systems, offering a practical tool for safety‑relevant applications.
Hugging Face Trending Papers
Jun 25
Structure Before Collapse: Transient semantic geometry in next-token prediction
Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other; a symmetric configuration that depends only on the output label and ignores any semantic similarity in the inputs. This creates a puzzle: next-token prediction language models are trained predominantly (as context length increases) with one-hot labels: the same context is very unlikely to appear twice in training with different labels.
arXiv Machine Learning
Sep 11
A Fragility Spectrum for Recursive Language-Model Training
The paper investigates how recursive contamination—retraining language models on their own generated text—affects output diversity across 13 publicly released checkpoints. Using a fixed contamination protocol over five generations, the authors find a wide spread in 4‑gram diversity (0.187 to 0.940), indicating that some models collapse into repetitive fragments while others remain largely unaffected. The study shows that a model’s susceptibility to collapse is an intrinsic property of the checkpoint, not predicted by parameter scale or static indicators, and that simple interventions such as tightening top‑p sampling can significantly slow or halt collapse.
arXiv AI
Sep 4
The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priors
The paper identifies a single direction in the unembedding matrix of large language models that encodes the unigram distribution of the training corpus, acting as a Bayesian prior when the model is uncertain. By projecting the final prediction state onto this direction, the authors derive a per‑token prior loading factor, λ, which decreases as context becomes more informative and decomposes predictions into a tempered prior and a context‑driven likelihood. Experiments across four model families (Llama, Qwen, Gemma, Pythia) show that larger models rely less on the prior in high‑context settings and that manipulating λ can steer predictions toward or away from the unigram prior in KL divergence.
arXiv Computation and Language
Aug 25
DynHD: Hallucination Detection for Diffusion Large Language Models via Denoising Dynamics Deviation Learning
DynHD is a method for detecting hallucinations in diffusion large language models (D‑LLMs) by focusing on token‑level uncertainty and its evolution during the denoising process. It introduces a semantic‑aware evidence construction module that filters out non‑informative structural tokens and highlights uncertainty in informative tokens, and a reference evidence generator that models the expected trajectory of uncertainty, enabling a deviation‑based detector to identify hallucinations. Experiments show DynHD outperforms existing baselines while being more efficient across various benchmarks and backbone models.