The paper investigates how autoregressive language models can iterate a hidden state without emitting tokens, identifying three distinct risks associated with a latent process: token risk, teacher‑forced latent risk, and open‑loop risk. It shows that when the Bayes next‑token posterior depends on a full‑rank $k$‑dimensional statistic, a width‑$k$ latent with linear least‑squares transition and a refitted unembedding can match the Bayes token predictor, but this match is only up to an invertible gauge. The study also highlights sharp obstructions: a constant latent can drive teacher‑forced risk to zero while token cross‑entropy reverts to marginal entropy, and a perfect token predictor may still conceal a null‑space walk that causes open‑loop error to grow linearly with horizon, indicating that small latent or token losses alone do not guarantee accurate latent rollout without additional rank or injectivity conditions.
arXiv:2609.36227v1 Announce Type: cross Abstract: Next-latent prediction fits a map from the current embedding to the next one. LeNEPA carries this objective to time series, replacing the stop-gradie...
arXiv:2608. 15798v1 Announce Type: new Abstract: Language models are compared by their held-out per-token cross-entropy risk---the quantity scaling laws are fitted to.
The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.
arXiv:2606. 07559v2 Announce Type: replace-cross Abstract: Fine-tuning a language model often fails silently when its correct completion must outrank a near-synonym competitor.
The paper investigates how entropy over chain‑of‑thought tokens influences policy decisions such as gradient application, pruning, and collapse detection. By separating scaffold tokens from substantive content, the authors analyze entropy, Kullback–Leibler divergence, and entropy velocity for each channel, proving differences between raw and content conventions and bounding answer diversity. Empirical results across 23 configurations show that scaffold tokens can account for up to 41% of high‑entropy positions, with entropy share growing through distillation, while content conventions outperform raw surprisal on compression tasks and reveal significant answer leakage in re‑fed chains.
arXiv:2609.38536v1 Announce Type: cross Abstract: Diffusion-based large language models (dLLMs) promise to break the sequential latency bottleneck of autoregressive agents through parallel decoding,...
arXiv:2602.02133v3 Announce Type: replace-cross Abstract: Autoregressive language models (ARMs) suffer from the reversal curse: after learning ''$A$ is $B$,'' they often fail on the reverse query ''$...
arXiv:2608.20647v1 Announce Type: new Abstract: Splitting a bidirectional LSTM's contextual representation into a forward-only $F_i$ (strictly a function of tokens $1..i$) and a backward-only $B_i$ (...
arXiv:2610.20104v1 Announce Type: new Abstract: Autoregressive transformers trained on limit order book (LOB) message streams are increasingly proposed as foundation models for market microstructure....
arXiv:2607. 06925v1 Announce Type: new Abstract: Compact world models that condition on a language goal promise to ground relations such as ``put the red block left of the blue block'' using a sparse set of explicit \emph{reference anchors}.
arXiv:2608. 06243v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models using automatically verifiable outcome signals, but these signals are typically sparse and at the sequence-level.
arXiv:2610.14007v1 Announce Type: cross Abstract: A familiar line of reasoning runs as follows: language models are flexible approximators trained under a strictly proper loss, so scale brings them c...