arXiv Machine Learning August 20, 2026 By Zachary Speck, Asa Shepard

Learned, Then Lost: A Measured Single-Example Counterfactual in Pre-training

Read the original on arXiv Machine Learning →

The study measured the impact of a single training example on a GPT‑2 model by running 24 counterfactual experiments. 32 models were trained from scratch on OpenWebText, and at a specific training step a single batch row was replaced with a 194‑token passage under three conditions (fluent prose, fabricated subject, random characters) or left unchanged. Results showed that the passage was learned from one exposure and decayed, with measurable differences in cross‑entropy up to 50 steps after injection but no lasting effect at the final step.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 22

The Information Shadow: Measuring Structural Limits on What Language Models Can Learn

arXiv:2607. 18305v1 Announce Type: cross Abstract: Some limits on what language models know are not gaps in data coverage but structural properties of learning from text.

By Priyansh Srivastava, Romit Chatterjee

llms benchmarks safety

arXiv Machine Learning

Oct 3

Why Does Train-Validation Separation Emerge? Update-Pressure Density Dynamics in Pretrained Backbones

The paper investigates why the train‑validation performance gap widens during fine‑tuning of pretrained models. It proposes a dynamic structural explanation: as training proceeds, updates shift from broadly reusable features to more example‑specific ones, increasing gradient heterogeneity and the gap. Experiments on synthetic ResMLP hierarchies, NLP models (RoBERTa, DeBERTa, Qwen) across six datasets, and vision models (ResNet‑18) confirm that higher reliance on private features correlates with larger accuracy gaps, supporting the proposed account.

By Yuchen Li, Mingyu Du, Zongqi Fan, Ken-Tye Yong, Nguyen H. Tran

nlp

arXiv Machine Learning

Aug 12

Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So

arXiv:2608. 10251v1 Announce Type: cross Abstract: A transformer's answer lives on one axis: the direction its unembedding reads.

By Mark Oskin

llms benchmarks

arXiv Computation and Language

Sep 28

Where a Model Sends Its Own Repeated Token

arXiv:2609. 31181v1 Announce Type: new Abstract: Black-box model identification works by scoring a model's response to natural-language prompts.

By Nicol\'as Vera Z\'u\~niga

nlp robotics

arXiv AI

Jul 9

Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix

arXiv:2607. 06925v1 Announce Type: new Abstract: Compact world models that condition on a language goal promise to ground relations such as ``put the red block left of the blue block'' using a sparse set of explicit \emph{reference anchors}.

By Yufeng Wang, Lu Wei, Haibin Ling

benchmarks

arXiv AI

Jun 9

Stage-1 Controls the Entropy Regime, Not the Outcome

arXiv:2606. 09059v1 Announce Type: cross Abstract: Two-stage post-training -- a Stage-1 warm-start (supervised fine-tuning, SFT, or on-policy distillation, OPD) followed by Stage-2 reinforcement learning (RL) -- is increasingly used for vision-language models (VLMs).

By Jianxiong Shen

llms reinforcement-learning fine-tuning efficiency multimodal

One email a morning, machine-written

Related stories

The Information Shadow: Measuring Structural Limits on What Language Models Can Learn

Why Does Train-Validation Separation Emerge? Update-Pressure Density Dynamics in Pretrained Backbones

Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So

Where a Model Sends Its Own Repeated Token

Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix

Stage-1 Controls the Entropy Regime, Not the Outcome