arXiv Computation and Language
Test-Time Training with Next-Token Prediction
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Jul 13
Self-Guided Test-Time Training for Long-Context LLMs
arXiv:2607. 09415v1 Announce Type: cross Abstract: Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs.
arXiv Machine Learning
Aug 4
Learning What to Remember: Test-Time Training via Context Distillation
arXiv:2608. 01672v1 Announce Type: cross Abstract: Effective long-context modeling is not merely about retaining more of the past, but about preserving the information that may prove relevant later.
arXiv Machine Learning
Aug 24
Rethinking Expressivity and Efficiency in Test-Time Training
arXiv:2608.21308v1 Announce Type: new Abstract: Test-Time Training (TTT) enables long-context processing via continuous weight updates during inference, but current methods struggle to balance the ex...
arXiv AI
Jul 24
SR-TTT Does Not Learn Retrieval: A Correction and Mechanistic Post-Mortem of Surprisal-Aware Residual Test-Time Training
arXiv:2603. 06642v2 Announce Type: replace-cross Abstract: Test-Time Training (TTT) language models replace the KV-cache with fast weights updated during inference, achieving O(1) memory but suffering catastrophic failure on exact-recall tasks.
arXiv Computation and Language
Sep 11
NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
NCP-ArchPreview is a latent‑space language model that extends standard next‑token prediction (NTP) with a Next Concept Prediction (NCP) objective, allowing the model to predict discrete concepts spanning multiple tokens. The architecture builds a product‑quantized concept vocabulary from hidden states, uses a dedicated Concept Module to forecast future concepts, and feeds these predictions back to guide token‑level generation, all trained jointly end‑to‑end. Trained on 5.73 T tokens with 8.9 B parameters, it achieves the final pretraining loss of OLMo‑3‑7B using only 51.3 % of the tokens, outperforms OLMo‑3‑7B on downstream tasks (including a 5.99‑point GSM8K gain), and demonstrates that the learned latent space enables lightweight domain adaptation and improved drafting performance.
arXiv Computation and Language
Sep 10
Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding
Osprey is a target‑agnostic pre‑training method that bootstraps draft models for speculative decoding from existing small language models. By pruning to a shallow backbone, restoring language‑modeling capability with next‑token pretraining, and adapting via vocabulary alignment and distillation, Osprey reduces per‑target work to a lightweight adaptation step. Experiments show that a single Osprey backbone improves mean acceptance length by up to 22.7% and increases tokens per second by 17.5% across several large target models, especially on out‑of‑domain and multilingual data.