arXiv Machine Learning
Conditional Transfer from Controlled Pretraining Mixtures to Code
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Oct 1
A helps B while B hurts A: directed transfer in instruction-tuning mixture
arXiv:2609.39702v1 Announce Type: new Abstract: Adapting a language model to a specialized corpus means choosing which instruction-tuning tasks to train on under a fixed budget, and testing one choic...
arXiv Computation and Language
Sep 10
Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding
Osprey is a target‑agnostic pre‑training method that bootstraps draft models for speculative decoding from existing small language models. By pruning to a shallow backbone, restoring language‑modeling capability with next‑token pretraining, and adapting via vocabulary alignment and distillation, Osprey reduces per‑target work to a lightweight adaptation step. Experiments show that a single Osprey backbone improves mean acceptance length by up to 22.7% and increases tokens per second by 17.5% across several large target models, especially on out‑of‑domain and multilingual data.
arXiv AI
Jun 15
Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining
arXiv:2601. 22108v2 Announce Type: replace-cross Abstract: Continued pretraining is optimized with fixed self-supervised tasks but selected by downstream performance, creating a coarse feedback loop in which practitioners evaluate checkpoints, change data mixtures or objectives, and restart runs, while individual updates remain blind to target capabilities.
arXiv AI
Jul 28
DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning
arXiv:2607. 22769v1 Announce Type: cross Abstract: The training efficacy of large language models (LLMs) is fundamentally constrained by the quality and composition of training data.
arXiv Computer Vision
Sep 10
BrainTaskonomy: Learning How to Pretrain and What to Transfer in fMRI Foundation Models
arXiv:2609.10518v1 Announce Type: new Abstract: fMRI foundation models increasingly aggregate heterogeneous data across brain states, cohorts, and acquisition settings, yet pretraining domains are co...
arXiv AI
Aug 11
How Far Do Foundation Models Transfer to Infant Signals? A Cross-Dataset Transfer Audit with a Unified Need Ontology
arXiv:2608. 08989v1 Announce Type: cross Abstract: Public infant cry corpora are small, label-incompatible, and almost always evaluated one corpus at a time.