arXiv Machine Learning August 20, 2026 By Zachary Speck, Asa Shepard

Learned, Then Lost: A Measured Single-Example Counterfactual in Pre-training

Read the original on arXiv Machine Learning →

The study measured the impact of a single training example on a GPT‑2 model by running 24 counterfactual experiments. 32 models were trained from scratch on OpenWebText, and at a specific training step a single batch row was replaced with a 194‑token passage under three conditions (fluent prose, fabricated subject, random characters) or left unchanged. Results showed that the passage was learned from one exposure and decayed, with measurable differences in cross‑entropy up to 50 steps after injection but no lasting effect at the final step.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 22

The Information Shadow: Measuring Structural Limits on What Language Models Can Learn

arXiv:2607. 18305v1 Announce Type: cross Abstract: Some limits on what language models know are not gaps in data coverage but structural properties of learning from text.

By Priyansh Srivastava, Romit Chatterjee

llms benchmarks safety

arXiv Machine Learning

Aug 12

Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So

arXiv:2608. 10251v1 Announce Type: cross Abstract: A transformer's answer lives on one axis: the direction its unembedding reads.

By Mark Oskin

llms benchmarks

arXiv AI

Jul 9

Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix

arXiv:2607. 06925v1 Announce Type: new Abstract: Compact world models that condition on a language goal promise to ground relations such as ``put the red block left of the blue block'' using a sparse set of explicit \emph{reference anchors}.

By Yufeng Wang, Lu Wei, Haibin Ling

benchmarks

arXiv AI

Jun 9

Stage-1 Controls the Entropy Regime, Not the Outcome

arXiv:2606. 09059v1 Announce Type: cross Abstract: Two-stage post-training -- a Stage-1 warm-start (supervised fine-tuning, SFT, or on-policy distillation, OPD) followed by Stage-2 reinforcement learning (RL) -- is increasingly used for vision-language models (VLMs).

By Jianxiong Shen

llms reinforcement-learning fine-tuning efficiency multimodal

arXiv Machine Learning

4d ago

You Only Pass Once: Answering and Abstaining Together in a Single Forward Pass of a Frozen Language Model

arXiv:2608. 14465v1 Announce Type: cross Abstract: A frozen language model on reasoning tasks has two coupled weaknesses: it under-uses evidence its own residual stream already encodes, and it fails to detect when the input is insufficient to answer, so it confabulates.

By Ziyang Luo, Zhongyao Chu, Xinjie He, Youting Wang, Xukui Qin, Runxiong Wu, Yan-Syuan Chen

llms benchmarks

arXiv AI

2d ago

Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models

arXiv:2608. 17564v1 Announce Type: cross Abstract: Unified multimodal models (UMMs) are motivated by the hope that understanding and generation reinforce each other but controlled ablations repeatedly find that adding a generation objective leaves understanding flat.

By Zongyang Qiu, Yihan Wu, Kaixuan Fan, Bo Li, Hui Xiong

diffusion multimodal safety

One email a morning, machine-written

Related stories

The Information Shadow: Measuring Structural Limits on What Language Models Can Learn

Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So

Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix

Stage-1 Controls the Entropy Regime, Not the Outcome

You Only Pass Once: Answering and Abstaining Together in a Single Forward Pass of a Frozen Language Model

Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models