Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training.
arXiv:2609.36569v1 Announce Type: cross
Abstract: Checkpoint selection is a routine decision in supervised fine-tuning (SFT): training produces multiple checkpoints, but only one is retained. Yet fix...
By Yupeng Chang, Wenxuan Zhang, Yuan Wu
arXiv:2609.37169v1 Announce Type: cross
Abstract: Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since add...
By Zhehao Huang, Changxin Tian, Qingyuan Yang, Kunlong Chen, Ziqi Liu, Zhiqiang Zhang, Xiaolin Huang, Jun Zhou
arXiv:2606. 04272v1 Announce Type: new Abstract: The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT).
By Rachit Bansal, Clara Mohri, Tian Qin, David Alvarez-Melis, Sham Kakade
arXiv:2607. 09204v1 Announce Type: cross Abstract: Pretrained language models often exhibit structured weight spectra, suggesting that training may repeatedly produce similar layerwise and component-wise organization.
By Konstantin Garbers, Nicholas Oh
The paper investigates how a probe can decode in‑context bindings on model errors and how probe‑guided steering can repair them. It tracks probe accuracy, model output, and steering response across public pretraining and post‑training checkpoints, noting that probe accuracy improves during Pythia pretraining and that steering benefits grow with model size. The study also shows that decoders trained on final state or candidate logits do not outperform each other on late‑checkpoint errors, and presents an information‑theoretic counterexample explaining why decodability on errors alone cannot prove discarded output information.
By Manas Venkata Sai Ravulapalli, Samrath Singh Chadha
arXiv:2608. 10090v1 Announce Type: new Abstract: Large language models (LLMs) have advanced code generation, where executable feedback provides a more reliable learning signal than textual imitation alone.
By Hejia Zhang, Sheng Lu, Zhongming Yu, Chia-Tung Ho, Brucek Khailany, Jishen Zhao
The paper investigates how different expert‑deactivation strategies at inference time—removal, freezing, remaining‑expert renormalization, and unnormalized re‑routing—affect token routing and expert weights. Experiments across five instruction‑tuned checkpoints from four model families, plus a base OLMoE checkpoint, show that the choice of strategy interacts with the training‑time weight rule: renormalization performs best when models were trained with raw router probabilities, while unnormalized re‑routing is superior when trained with normalized weights. A weight‑sum model links loss of expert contributions and branch‑scale changes to the training rule, revealing that mismatches can inflate perplexity by up to 28,740× and that additional renormalization penalties grow as fewer experts are activated.
By Hao Cheng (Zhejiang Gongshang University), Zhiyu Yang (Chang'an University), Bo Ma (Zhejiang Gongshang University), Hanzhang Jia (University of Sussex)
Temporal models that maintain per-sequence latent states during training must reconstruct those states when the stored table is unavailable at inference. This creates a basic consistency question: do...
UpgradeBench is a decision‑centric longitudinal benchmark that evaluates how fine‑tuned language‑model specialists should be handled when new base‑model releases occur. It covers four consecutive Qwen releases, a continuation checkpoint, six tasks, two model sizes, and OLMo checkpoints with known training lineage, and examines whether retraining, adapter transfer, or other recovery strategies improve specialist performance. The benchmark reveals that upgrade gains vary by task and release interval, that direct adapter copying is sensitive to pretraining distance, and that teacher relabeling can recover specialists without new annotations.
"whyItMatters":"The study provides actionable insights into the cost‑effective management of specialist models across model releases, showing how to balance retraining effort with performance gains."
By Ye Chen, Weining Zhang
arXiv:2608. 14929v1 Announce Type: cross Abstract: Open-weight language models are fine-tuned, quantized, pruned, and merged, yet their provenance is often undocumented.
By Aman Singh Thakur, Rayan Khoury
arXiv:2607. 25063v1 Announce Type: new Abstract: Developers judge a model checkpoint by how it behaves.
By Cen Lu, Yung-Chen Tang, Andrea Cavallaro