arXiv Machine Learning
Tracing Morphology, Semantics, and Divergence in Language Acquisition
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computation and Language
Oct 12
Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
Large language models show strong cross‑lingual abilities, but previous studies have examined them only at isolated training points. This work pretrains a 1.7B multilingual model on nine languages, recording checkpoints at fine granularity to track translation development. Using word‑level translation and a new dataset, the authors find that early learning is dominated by token‑level copying and surface similarities, while later stages involve more general translation mechanisms and refined copying.
arXiv Machine Learning
Aug 31
Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss
The paper investigates how learning rate and batch size scale when pretraining dense large language models on English‑prevalent corpora, examining both jointly optimal and marginal evolutions across model capacity and data size. It explores the benefits of a Warmup‑Stable‑Decay learning‑rate schedule, assessing whether optimal hyperparameters transfer between stable and decay phases, and evaluates loss scaling forms that capture interactions between model capacity and dataset size. The study provides a baseline scaling procedure and releases the full set of pretraining runs for future OpenEuroLLM development.
arXiv Machine Learning
Oct 3
Persistent Depth Ordering amid Shifting Block-Bypass Responses in Language Model Pretraining
The study investigates how layer‑wise intervention responses in language models change over the course of pretraining, using single‑block identity bypass across multiple checkpoints and model‑domain combinations. It finds that while depth ordering of responses persists, their magnitudes shift, with nearby checkpoints showing stronger rank correspondence than distant ones and large changes occurring at positions that recur across samples and transfer across evaluation domains. Controlled experiments reveal that these longitudinal changes cannot be explained by a single downstream sensitivity and depend on perturbation strength and direction, indicating that layer sensitivity is structured but dynamic.
arXiv AI
5d ago
Learning to Read Out: Unembedding Dynamics in Language Model Pretraining
The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.
arXiv Machine Learning
Sep 1
Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization
arXiv:2608.30315v1 Announce Type: new Abstract: Token embeddings are the basic representational units that connect discrete tokens with continuous computation in language models. Although modern lang...
arXiv Machine Learning
Oct 7
A theory of platonic representations in language models
arXiv:2610.07168v1 Announce Type: new Abstract: Representations of translated sentences are similar in the inner layers of multilingual language models -- an observation connected to the platonic rep...