← Back to all news
arXiv AI By Xiaohong Chen, David Bucur, Chenglong Ma, Yi Zhang, Lingming Zhang, Sriram Vishwanath, Grigore Rosu

Evaluating Exact Output and Checkpoint-State Prediction in Real Programs

Read the original on arXiv AI →

The paper introduces a benchmark for predicting a program’s final output and checkpoint state using only its source code and input, extending CRUXEval-style tasks with both short‑ and long‑trace inputs and checkpoints inside and after loops. It evaluates 400 cases from 371 Python and C++ programs across seven settings from four model families, achieving up to 93.0% accuracy on short‑trace final output and 77.0% on long‑trace final output, with reasoning‑enabled settings outperforming non‑reasoning ones by 33.1 to 55.2 percentage points. The study also shows that longer‑trace inputs can change correctness outcomes and that the benchmark reveals errors that short‑output scores alone would miss.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 28

Checkpoint Reload Explains Most of the Measured State Gap in Two State Discovery Reproductions

The paper investigates whether the latent states reconstructed during inference match those stored during training in temporal models, focusing on State Discovery in World Machine. By comparing stored state tables with rollouts from the best‑validation checkpoint and the last‑epoch weights, the authors find that reloading the checkpoint used to write the table eliminates 90–91% of the mean per‑dimension Wasserstein‑1 distance, explaining most of the observed state gap. Causal distillation and frozen SD updates fail to meet their criteria under either checkpoint.

By Weijia Han, Lisha Qu
efficiency
More like this →
arXiv Machine Learning
Sep 29

Decodable In-Context State and Model Output Across Training

The paper investigates how a probe can decode in‑context bindings on model errors and how probe‑guided steering can repair them. It tracks probe accuracy, model output, and steering response across public pretraining and post‑training checkpoints, noting that probe accuracy improves during Pythia pretraining and that steering benefits grow with model size. The study also shows that decoders trained on final state or candidate logits do not outperform each other on late‑checkpoint errors, and presents an information‑theoretic counterexample explaining why decodability on errors alone cannot prove discarded output information.

By Manas Venkata Sai Ravulapalli, Samrath Singh Chadha
More like this →