Predictive Latents for Autoregressive Language Models: Possibility, Gauge, and Obstruction

The paper investigates how autoregressive language models can iterate a hidden state without emitting tokens, identifying three distinct risks associated with a latent process: token risk, teacher‑forced latent risk, and open‑loop risk. It shows that when the Bayes next‑token posterior depends on a full‑rank $k$‑dimensional statistic, a width‑$k$ latent with linear least‑squares transition and a refitted unembedding can match the Bayes token predictor, but this match is only up to an invertible gauge. The study also highlights sharp obstructions: a constant latent can drive teacher‑forced risk to zero while token cross‑entropy reverts to marginal entropy, and a perfect token predictor may still conceal a null‑space walk that causes open‑loop error to grow linearly with horizon, indicating that small latent or token losses alone do not guarantee accurate latent rollout without additional rank or injectivity conditions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

• llms