The paper investigates whether the latent states reconstructed during inference match those stored during training in temporal models, focusing on State Discovery in World Machine. By comparing stored state tables with rollouts from the best‑validation checkpoint and the last‑epoch weights, the authors find that reloading the checkpoint used to write the table eliminates 90–91% of the mean per‑dimension Wasserstein‑1 distance, explaining most of the observed state gap. Causal distillation and frozen SD updates fail to meet their criteria under either checkpoint.
By Weijia Han, Lisha Qu
The paper investigates how a probe can decode in‑context bindings on model errors and how probe‑guided steering can repair them. It tracks probe accuracy, model output, and steering response across public pretraining and post‑training checkpoints, noting that probe accuracy improves during Pythia pretraining and that steering benefits grow with model size. The study also shows that decoders trained on final state or candidate logits do not outperform each other on late‑checkpoint errors, and presents an information‑theoretic counterexample explaining why decodability on errors alone cannot prove discarded output information.
By Manas Venkata Sai Ravulapalli, Samrath Singh Chadha
The audit examines 263 batch‑normalized checkpoints released by arXiv, finding that refitting models on retained data at identical weights shifts 47 of 221 checkpoints beyond the spread indicated by their own release seeds. This movement is attributed to checkpoint properties rather than the survival of removed data, as swapping removed records for kept ones barely changes the published state. The study concludes that releases should specify the fitting convention used, especially for batch‑normalized vision models.
By Junlong Shen Xingyu Li
ReplaySSM is a technique that redefines how state-space models (SSMs) handle their recurrent state by caching SSM inputs instead of the state itself. It unrolls the recurrence into a checkpoint plus a buffer of recent inputs, reducing state memory traffic by about half and enabling efficient speculative rollback. The method also allows separate precision levels for checkpoints, improving output accuracy while accelerating decoding in hybrid models such as Mamba-2 and Gated DeltaNet.
By Ze-Wei Liou, Harris Nover, Amey Naik, Andrii Skliar, Igor Shovkun, Ido Shahaf, Tri Dao
The paper introduces a benchmark for predicting a program’s final output and checkpoint state using only its source code and input, extending CRUXEval-style tasks with both short‑ and long‑trace inputs and checkpoints inside and after loops. It evaluates 400 cases from 371 Python and C++ programs across seven settings from four model families, achieving up to 93.0% accuracy on short‑trace final output and 77.0% on long‑trace final output, with reasoning‑enabled settings outperforming non‑reasoning ones by 33.1 to 55.2 percentage points. The study also shows that longer‑trace inputs can change correctness outcomes and that the benchmark reveals errors that short‑output scores alone would miss.
By Xiaohong Chen, David Bucur, Chenglong Ma, Yi Zhang, Lingming Zhang, Sriram Vishwanath, Grigore Rosu
arXiv:2609.36569v1 Announce Type: cross
Abstract: Checkpoint selection is a routine decision in supervised fine-tuning (SFT): training produces multiple checkpoints, but only one is retained. Yet fix...
By Yupeng Chang, Wenxuan Zhang, Yuan Wu
arXiv:2508. 12220v2 Announce Type: replace-cross Abstract: Can a prospectively instrumented training continuation reproduce a deletion counterfactual exactly after selected examples leave its replay dataset?
By Abdullah X
The paper introduces OrbitTrace, a benchmark of 50 physics‑grounded compute‑availability traces from satellite orbits, and investigates whether specialized interruption‑resilient optimizers are needed when training is interrupted by predictable compute gaps. Experiments on CIFAR‑10/ResNet‑18 and GPT‑2/AdamW show that a strong checkpoint‑and‑resume baseline that preserves full optimizer state and indexes learning‑rate schedules in effective time matches uninterrupted training, rendering most availability‑aware methods unnecessary. Only in a narrow regime—large models with non‑persistable optimizer state and frequent short pauses—does reactive adaptation recover a modest portion of the state‑loss penalty, and even this benefit disappears for eclipse‑scale gaps.
By Subhadip Mitra
arXiv:2609.06872v1 Announce Type: new
Abstract: When a user asks an assistant to forget a record, the test is whether the memory now matches the state it would hold if the record had never been store...
By Vishwajith Ramesh
arXiv:2609.01048v1 Announce Type: cross
Abstract: Across the full Pythia suite (160M-12B, eight checkpoints, four task families), a linear probe can read a target variable from the residual stream as...
By Xining Xun
arXiv:2610.20071v1 Announce Type: new
Abstract: MeanFlow predicts interval-average velocities, but conventional few-step sampling ties the interval queried from the model to the duration of the state...
By Tinglan Wang, Haixu Liu, Shaobo Chen, Nuo Qian, Qi Gao, Kai Wen, Yu Yao
arXiv:2609.08966v1 Announce Type: new
Abstract: Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the...
By Sohir Maskey, Philipp Scholl, Jonas Knupp, Pit Neitemeier, Sascha Wirges