arXiv:2609.39702v1 Announce Type: new Abstract: Adapting a language model to a specialized corpus means choosing which instruction-tuning tasks to train on under a fixed budget, and testing one choic...
Osprey is a target‑agnostic pre‑training method that bootstraps draft models for speculative decoding from existing small language models. By pruning to a shallow backbone, restoring language‑modeling capability with next‑token pretraining, and adapting via vocabulary alignment and distillation, Osprey reduces per‑target work to a lightweight adaptation step. Experiments show that a single Osprey backbone improves mean acceptance length by up to 22.7% and increases tokens per second by 17.5% across several large target models, especially on out‑of‑domain and multilingual data.
arXiv:2601. 22108v2 Announce Type: replace-cross Abstract: Continued pretraining is optimized with fixed self-supervised tasks but selected by downstream performance, creating a coarse feedback loop in which practitioners evaluate checkpoints, change data mixtures or objectives, and restart runs, while individual updates remain blind to target capabilities.
arXiv:2607. 22769v1 Announce Type: cross Abstract: The training efficacy of large language models (LLMs) is fundamentally constrained by the quality and composition of training data.
arXiv:2609.10518v1 Announce Type: new Abstract: fMRI foundation models increasingly aggregate heterogeneous data across brain states, cohorts, and acquisition settings, yet pretraining domains are co...
arXiv:2608. 08989v1 Announce Type: cross Abstract: Public infant cry corpora are small, label-incompatible, and almost always evaluated one corpus at a time.
arXiv:2609.37169v1 Announce Type: cross Abstract: Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since add...
arXiv:2605. 12705v2 Announce Type: replace Abstract: How can we train models whose post-trained capabilities survive subsequent fine-tuning?
arXiv:2607. 01104v1 Announce Type: cross Abstract: In Large Language Model (LLM) training, data mixing plays a pivotal role in determining model performance.
arXiv:2609.16229v1 Announce Type: new Abstract: Machine unlearning aims to remove specific knowledge from a trained large language model (LLM) without retraining from scratch. Existing methods modify...
arXiv:2604. 26170v2 Announce Type: replace Abstract: Adapting large language models (LLMs) to a targeted task efficiently and effectively remains a fundamental challenge.
The paper compares two low‑budget methods for converting a 30B Mixture‑of‑Experts autoregressive language model into a diffusion language model. One method updates a subset of the model’s weights in‑place, while the other freezes the context tower and conditions on a frozen causal copy via cross‑attention. With only 1B training tokens, the frozen‑tower approach achieves a HumanEval pass@10 score of 71.60 versus 6.19 for the in‑place method, and retains 95% of the parent’s GSM8K and 99% of its MMLU‑Pro performance.