The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The paper investigates whether the latent states reconstructed during inference match those stored during training in temporal models, focusing on State Discovery in World Machine. By comparing stored state tables with rollouts from the best‑validation checkpoint and the last‑epoch weights, the authors find that reloading the checkpoint used to write the table eliminates 90–91% of the mean per‑dimension Wasserstein‑1 distance, explaining most of the observed state gap. Causal distillation and frozen SD updates fail to meet their criteria under either checkpoint.
The paper investigates how a probe can decode in‑context bindings on model errors and how probe‑guided steering can repair them. It tracks probe accuracy, model output, and steering response across public pretraining and post‑training checkpoints, noting that probe accuracy improves during Pythia pretraining and that steering benefits grow with model size. The study also shows that decoders trained on final state or candidate logits do not outperform each other on late‑checkpoint errors, and presents an information‑theoretic counterexample explaining why decodability on errors alone cannot prove discarded output information.
The audit examines 263 batch‑normalized checkpoints released by arXiv, finding that refitting models on retained data at identical weights shifts 47 of 221 checkpoints beyond the spread indicated by their own release seeds. This movement is attributed to checkpoint properties rather than the survival of removed data, as swapping removed records for kept ones barely changes the published state. The study concludes that releases should specify the fitting convention used, especially for batch‑normalized vision models.
ReplaySSM is a technique that redefines how state-space models (SSMs) handle their recurrent state by caching SSM inputs instead of the state itself. It unrolls the recurrence into a checkpoint plus a buffer of recent inputs, reducing state memory traffic by about half and enabling efficient speculative rollback. The method also allows separate precision levels for checkpoints, improving output accuracy while accelerating decoding in hybrid models such as Mamba-2 and Gated DeltaNet.
The paper introduces a benchmark for predicting a program’s final output and checkpoint state using only its source code and input, extending CRUXEval-style tasks with both short‑ and long‑trace inputs and checkpoints inside and after loops. It evaluates 400 cases from 371 Python and C++ programs across seven settings from four model families, achieving up to 93.0% accuracy on short‑trace final output and 77.0% on long‑trace final output, with reasoning‑enabled settings outperforming non‑reasoning ones by 33.1 to 55.2 percentage points. The study also shows that longer‑trace inputs can change correctness outcomes and that the benchmark reveals errors that short‑output scores alone would miss.
arXiv:2609.36569v1 Announce Type: cross Abstract: Checkpoint selection is a routine decision in supervised fine-tuning (SFT): training produces multiple checkpoints, but only one is retained. Yet fix...