Temporal models that maintain per-sequence latent states during training must reconstruct those states when the stored table is unavailable at inference. This creates a basic consistency question: do...
The audit examines 263 batch‑normalized checkpoints released by arXiv, finding that refitting models on retained data at identical weights shifts 47 of 221 checkpoints beyond the spread indicated by their own release seeds. This movement is attributed to checkpoint properties rather than the survival of removed data, as swapping removed records for kept ones barely changes the published state. The study concludes that releases should specify the fitting convention used, especially for batch‑normalized vision models.
By Junlong Shen Xingyu Li
The paper investigates how a probe can decode in‑context bindings on model errors and how probe‑guided steering can repair them. It tracks probe accuracy, model output, and steering response across public pretraining and post‑training checkpoints, noting that probe accuracy improves during Pythia pretraining and that steering benefits grow with model size. The study also shows that decoders trained on final state or candidate logits do not outperform each other on late‑checkpoint errors, and presents an information‑theoretic counterexample explaining why decodability on errors alone cannot prove discarded output information.
By Manas Venkata Sai Ravulapalli, Samrath Singh Chadha
The paper introduces a benchmark for predicting a program’s final output and checkpoint state using only its source code and input, extending CRUXEval-style tasks with both short‑ and long‑trace inputs and checkpoints inside and after loops. It evaluates 400 cases from 371 Python and C++ programs across seven settings from four model families, achieving up to 93.0% accuracy on short‑trace final output and 77.0% on long‑trace final output, with reasoning‑enabled settings outperforming non‑reasoning ones by 33.1 to 55.2 percentage points. The study also shows that longer‑trace inputs can change correctness outcomes and that the benchmark reveals errors that short‑output scores alone would miss.
By Xiaohong Chen, David Bucur, Chenglong Ma, Yi Zhang, Lingming Zhang, Sriram Vishwanath, Grigore Rosu
ReplaySSM is a technique that redefines how state-space models (SSMs) handle their recurrent state by caching SSM inputs instead of the state itself. It unrolls the recurrence into a checkpoint plus a buffer of recent inputs, reducing state memory traffic by about half and enabling efficient speculative rollback. The method also allows separate precision levels for checkpoints, improving output accuracy while accelerating decoding in hybrid models such as Mamba-2 and Gated DeltaNet.
By Ze-Wei Liou, Harris Nover, Amey Naik, Andrii Skliar, Igor Shovkun, Ido Shahaf, Tri Dao
arXiv:2609.36569v1 Announce Type: cross
Abstract: Checkpoint selection is a routine decision in supervised fine-tuning (SFT): training produces multiple checkpoints, but only one is retained. Yet fix...
By Yupeng Chang, Wenxuan Zhang, Yuan Wu