arXiv AI
Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Aug 13
Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining
Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training.
arXiv AI
1d ago
From Checkpoint Variation to Selection Gains in Supervised Fine-Tuning
arXiv:2609.36569v1 Announce Type: cross Abstract: Checkpoint selection is a routine decision in supervised fine-tuning (SFT): training produces multiple checkpoints, but only one is retained. Yet fix...
arXiv AI
1d ago
Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories
arXiv:2609.37169v1 Announce Type: cross Abstract: Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since add...
arXiv Machine Learning
Jun 4
RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
arXiv:2606. 04272v1 Announce Type: new Abstract: The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT).
arXiv Machine Learning
Jul 13
Complexity-Guided Component-wise Initialization for Language Model Pretraining
arXiv:2607. 09204v1 Announce Type: cross Abstract: Pretrained language models often exhibit structured weight spectra, suggesting that training may repeatedly produce similar layerwise and component-wise organization.
arXiv Machine Learning
2d ago
Decodable In-Context State and Model Output Across Training
The paper investigates how a probe can decode in‑context bindings on model errors and how probe‑guided steering can repair them. It tracks probe accuracy, model output, and steering response across public pretraining and post‑training checkpoints, noting that probe accuracy improves during Pythia pretraining and that steering benefits grow with model size. The study also shows that decoders trained on final state or candidate logits do not outperform each other on late‑checkpoint errors, and presents an information‑theoretic counterexample explaining why decodability on errors alone cannot prove discarded output information.