arXiv Machine Learning
Lasting Effects of Abstract Pretraining Beyond Perplexity
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
Hugging Face Trending Papers
Jun 8
Multi-Hop Knowledge Composition is Bound by Pretraining Exposure
Large Language Models fail at implicit multi-hop reasoning: a model answers "When was $X$ born? " and "Who is $Y$'s closest friend?
arXiv AI
4d ago
LLMs are not stochastic parrots: Evidence for meaning-mediated abstraction from conlang-like tasks
arXiv:2609.34187v2 Announce Type: replace-cross Abstract: The strong version of the stochastic parrot argument claims that, although large language models (LLMs) may exceed rote regurgitation, they c...
arXiv Computation and Language
Sep 17
Multi-Hop Knowledge Composition is Bound by Pretraining Exposure
Large Language Models struggle with implicit multi‑hop reasoning, correctly answering individual facts but failing to combine them in a single pass. In a controlled setting, the authors show that this failure persists even with high 1‑hop accuracy, indicating it is due to pretraining exposure rather than missing knowledge. They test nine data‑centric augmentation formats and find that only individuals seen in compositional contexts during pretraining enable transfer to unseen questions, proving exposure to such contexts is necessary for implicit multi‑hop reasoning.
arXiv AI
Jun 17
Small Initialization Matters for Large Language Models
arXiv:2606. 17945v1 Announce Type: new Abstract: Large language models provide a tractable system for asking how intelligence itself emerges, rather than only how LLMs can be engineered.
arXiv Machine Learning
4d ago
It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
arXiv:2609.37891v1 Announce Type: cross Abstract: Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--...
arXiv AI
Jul 20
Understanding Reasoning from Pretraining to Post-Training
arXiv:2607. 16097v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it.