arXiv:2607. 09415v1 Announce Type: cross Abstract: Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs.
arXiv:2608. 01672v1 Announce Type: cross Abstract: Effective long-context modeling is not merely about retaining more of the past, but about preserving the information that may prove relevant later.
The paper "Reinforced Fast Weights via Next-Sequence Prediction" introduces ReFINE, a method that extends the next-token prediction objective to next-sequence prediction for fast weight architectures. ReFINE selects informative token positions, generates multi-token rollouts, assigns sequence-level rewards, and optimizes the model using Group Relative Policy Optimization. Experiments on LaCT-760M and DeltaNet-1.3B demonstrate that ReFINE consistently outperforms standard next-token prediction fine‑tuning across retrieval, long‑context question answering, and LongBench tasks.
arXiv:2608.21308v1 Announce Type: new Abstract: Test-Time Training (TTT) enables long-context processing via continuous weight updates during inference, but current methods struggle to balance the ex...
arXiv:2610.00888v1 Announce Type: cross Abstract: Multi-token prediction (MTP) improves the throughput of autoregressive generation by enabling the language model to draft multiple next tokens per fo...
arXiv:2603. 06642v2 Announce Type: replace-cross Abstract: Test-Time Training (TTT) language models replace the KV-cache with fast weights updated during inference, achieving O(1) memory but suffering catastrophic failure on exact-recall tasks.
NCP-ArchPreview is a latent‑space language model that extends standard next‑token prediction (NTP) with a Next Concept Prediction (NCP) objective, allowing the model to predict discrete concepts spanning multiple tokens. The architecture builds a product‑quantized concept vocabulary from hidden states, uses a dedicated Concept Module to forecast future concepts, and feeds these predictions back to guide token‑level generation, all trained jointly end‑to‑end. Trained on 5.73 T tokens with 8.9 B parameters, it achieves the final pretraining loss of OLMo‑3‑7B using only 51.3 % of the tokens, outperforms OLMo‑3‑7B on downstream tasks (including a 5.99‑point GSM8K gain), and demonstrates that the learned latent space enables lightweight domain adaptation and improved drafting performance.
arXiv:2610.19210v1 Announce Type: new Abstract: Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference r...
The paper introduces CopyGen, a lightweight copy-constrained decoding plugin for frozen large language models that improves exact span reproduction. CopyGen decomposes copying into three decisions—when to copy, how far to copy, and when to stop—using adaptive logit adjustments during verification to emit the longest valid prefix. Across three LLM backbones and four benchmark tasks, it boosts average exact match by 35% relative without finetuning and further improves by 4% with SFT, while also accelerating generation.
arXiv:2601.22628v3 Announce Type: replace-cross Abstract: Test-Time Training offers a promising way to improve the reasoning ability of large language models (LLMs) by adapting the model using only t...
TurboMamba is a new language‑model architecture that combines a global Mamba‑3 model with a small Transformer to achieve faster inference, faster training, and better token prediction. It introduces three lightweight enhancements—direct access to the previous block’s last token, a small correction to next‑token scores, and a learned mixture with recent‑token frequencies—to improve local predictions. In experiments on FineWeb‑Edu, TurboMamba outperforms Transformer, Mamba‑3, Jamba, and Zamba, achieving the lowest mean test prediction loss and significantly higher training throughput and batch latency speedups.
Osprey is a target‑agnostic pre‑training method that bootstraps draft models for speculative decoding from existing small language models. By pruning to a shallow backbone, restoring language‑modeling capability with next‑token pretraining, and adapting via vocabulary alignment and distillation, Osprey reduces per‑target work to a lightweight adaptation step. Experiments show that a single Osprey backbone improves mean acceptance length by up to 22.7% and increases tokens per second by 17.5% across several large target models, especially on out‑of‑domain and multilingual data.