NCP-ArchPreview is a latent‑space language model that extends standard next‑token prediction (NTP) with a Next Concept Prediction (NCP) objective, allowing the model to predict discrete concepts spanning multiple tokens. The architecture builds a product‑quantized concept vocabulary from hidden states, uses a dedicated Concept Module to forecast future concepts, and feeds these predictions back to guide token‑level generation, all trained jointly end‑to‑end. Trained on 5.73 T tokens with 8.9 B parameters, it achieves the final pretraining loss of OLMo‑3‑7B using only 51.3 % of the tokens, outperforms OLMo‑3‑7B on downstream tasks (including a 5.99‑point GSM8K gain), and demonstrates that the learned latent space enables lightweight domain adaptation and improved drafting performance.
arXiv:2511. 05963v4 Announce Type: replace Abstract: Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens.
arXiv:2606. 10932v1 Announce Type: cross Abstract: We present Density Field State Space Models (DF-SSM), a framework for compressing SSMs to a 1-bit scaffold with int8 low-rank correction.
The paper "Reinforced Fast Weights via Next-Sequence Prediction" introduces ReFINE, a method that extends the next-token prediction objective to next-sequence prediction for fast weight architectures. ReFINE selects informative token positions, generates multi-token rollouts, assigns sequence-level rewards, and optimizes the model using Group Relative Policy Optimization. Experiments on LaCT-760M and DeltaNet-1.3B demonstrate that ReFINE consistently outperforms standard next-token prediction fine‑tuning across retrieval, long‑context question answering, and LongBench tasks.
arXiv:2608. 01672v1 Announce Type: cross Abstract: Effective long-context modeling is not merely about retaining more of the past, but about preserving the information that may prove relevant later.
arXiv:2610.25086v2 Announce Type: replace Abstract: Looped language models increase computational depth by repeatedly applying shared layers, offering a way to improve model quality without adding pa...
arXiv:2607. 18302v1 Announce Type: new Abstract: Autoregressive language models are least accurate at the beginning of a sequence, where little context forces reliance on a generic pretraining prior.
The study evaluates synthetic pre‑pretraining (PPT) across four curricula and three training regimes, finding that allocating compute to PPT never outperforms pure natural‑language training in compute‑matched experiments. Even when natural language data are scarce, repeated epochs over a small natural‑language corpus beat all synthetic curricula on validation loss, and no consistent downstream accuracy advantage is observed. The authors note that PPT gains are sensitive to weight initialization and that models tend to forget PPT knowledge during subsequent natural‑language training, suggesting limited benefits for PPT as a distinct training stage.
arXiv:2606.21803v2 Announce Type: replace Abstract: Next-token prediction is the self-supervised signal that trains language models, and every observed prompt token provides the same signal at test t...
arXiv:2606. 05173v1 Announce Type: cross Abstract: Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT, yet it encourages representations that are strongly anchored to surface-form token identity rather than deeper semantic structure.
arXiv:2511. 08577v3 Announce Type: replace-cross Abstract: Improving the reasoning abilities of Large Language Models (LLMs), especially under parameter constraints, is crucial for real-world applications.
arXiv:2608. 03494v1 Announce Type: cross Abstract: Vocabulary extension is an efficient way to adapt pretrained large language models (LLMs) to new languages, but the initialization of newly added token embeddings can strongly affect continued pre-training (CPT) efficiency.
Osprey is a target‑agnostic pre‑training method that bootstraps draft models for speculative decoding from existing small language models. By pruning to a shallow backbone, restoring language‑modeling capability with next‑token pretraining, and adapting via vocabulary alignment and distillation, Osprey reduces per‑target work to a lightweight adaptation step. Experiments show that a single Osprey backbone improves mean acceptance length by up to 22.7% and increases tokens per second by 17.5% across several large target models, especially on out‑of‑domain and multilingual data.