arXiv:2606. 27550v1 Announce Type: cross Abstract: Multi-token prediction has been shown to increase data density during training, improve downstream text-generation quality, and serves as the defacto approach for self-speculative decoding.
arXiv:2606. 14943v1 Announce Type: cross Abstract: Causal Transformers model sequences through an autoregressive factorization of the joint distribution, which enables efficient left-to-right decoding and conditional likelihood computation.
The paper "Reinforced Fast Weights via Next-Sequence Prediction" introduces ReFINE, a method that extends the next-token prediction objective to next-sequence prediction for fast weight architectures. ReFINE selects informative token positions, generates multi-token rollouts, assigns sequence-level rewards, and optimizes the model using Group Relative Policy Optimization. Experiments on LaCT-760M and DeltaNet-1.3B demonstrate that ReFINE consistently outperforms standard next-token prediction fine‑tuning across retrieval, long‑context question answering, and LongBench tasks.
arXiv:2608. 15592v1 Announce Type: new Abstract: Efficient LLM serving is often bottlenecked by the need to pad sequences to a fixed maximum length, and this wastes compute and degrades throughput.
The paper introduces CADENCE, a context‑adaptive decoding method that learns both what to draft and how far to draft during language generation. It uses Gaussian prediction heads to encode future token spans as continuous vectors, a shared decoder to produce tokens in parallel, and a learned router to choose the prediction length based on hidden states and Gaussian parameters. Experiments on Llama‑3.1‑8B‑Instruct show that CADENCE’s router‑selected positions achieve higher draft acceptance and up to 1.3× higher generation throughput compared to standard greedy decoding.
arXiv:2607. 17524v1 Announce Type: cross Abstract: We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task.
arXiv:2608.30627v1 Announce Type: new Abstract: As language-model compute continues to scale, high-quality training data is becoming an increasingly important bottleneck. Conventional next-token pred...
arXiv:2605. 12288v3 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions.
The article presents a mathematical framework for large language models (LLMs), detailing how text sequences are encoded into tokens, how next‑token prediction architectures are defined, and how these models are trained and deployed for tasks such as summarization, recommendation, software writing, and quantitative problem solving. It emphasizes that the framework relies on basic concepts from information theory, probability, and optimization, yet captures the complex algorithmic structure responsible for LLMs’ empirical successes. The authors argue that this formalism enables the study of accuracy, efficiency, and robustness, and points toward new methodological developments.
The paper proposes an information-weighted cross‑entropy loss that rescales token contributions using TF‑IDF statistics, thereby emphasizing semantically informative tokens and down‑weighting ubiquitous ones. Experiments on five decoder‑only language models (1.1B–13B parameters) show consistent reductions in memorized substring length while maintaining perplexity and downstream performance. The method is architecture‑agnostic, adds less than 3% computational overhead, and can be integrated into existing training pipelines.
arXiv:2608. 19529v1 Announce Type: cross Abstract: Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language.
Conceptizer is a training framework that augments next-token prediction with a reward for meaning by using a fixed vocabulary of about 9,000 concepts derived from contextual synonyms. It maps each token distribution to a concept distribution, sharing the reward for the target meaning across its possible wordings, thereby providing a concept-level training signal without adding model parameters. Applied to 1B–8B parameter models, Conceptizer achieves lower perplexity at concept-target positions, assigns higher probability to contextual synonyms, and improves downstream performance on knowledge and sentiment tasks while remaining comparable on commonsense tasks.