arXiv Machine Learning
Pre-carved Niches: The Formation Dynamics of Modular Task Partitions in Early LLM Training
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 7
Small Foundation Models of Human Cognition and Behaviour
arXiv:2608. 05224v1 Announce Type: new Abstract: Large language models fine-tuned on human behavioural data have emerged as general-purpose cognitive proxies, but the scale this requires, and whether these models process task structure or exploit statistical shortcuts, remain open questions.
arXiv AI
Sep 1
Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models
The study investigates how post‑training of large autoregressive language models (ARMs) into masked diffusion models (MDMs) affects their internal computation. Across two 7B ARM‑MDM families and four diagnostic tasks, the authors find that MDMs retain much of the ARM’s high‑attribution pathways on prefix‑dominant tasks, but reorganize computation toward earlier layers on globally constrained tasks. Component‑level probes reveal that ARMs depend on sharply specialized components, whereas MDMs show weaker specialization and more diffuse output‑space alignment.
arXiv AI
3d ago
Learning to Read Out: Unembedding Dynamics in Language Model Pretraining
The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.
arXiv AI
Oct 12
Task Context Shapes Component Roles in Language Models
The study investigates how task context reshapes the roles of shared attention heads in large language models. By applying identical graph facts and measuring interventions on attention heads and downstream activations, the authors show that task changes redistribute influence across five open‑weight models, causing the same heads to respond to different facts and affect later components differently. Linear response maps before and after activation edits reveal significant reductions in prediction error, indicating that task context alters both the facts processed and the downstream impact of shared components.
arXiv AI
Sep 30
Signatures of semantic search in the activations of large language models
arXiv:2609.35599v2 Announce Type: replace Abstract: When recalling lists of concepts (e.g., animals) during the semantic fluency task (SFT), both humans and large language models (LLMs) organise thei...
arXiv Machine Learning
Aug 4
Emergence Invariance: From Symbolized Thought to Interface Refinement
arXiv:2608. 01548v1 Announce Type: cross Abstract: Language can be viewed as a formalized subset of thought: a consequence-governed symbolic structure projected from wider situated cognition.