RW‑TTT addresses the challenge of serving large language models (LLMs) that perform test‑time training (TTT) by reading and updating request‑owned state, such as fast weights or streaming learner state. The method tags each decoding step with its owner, version, and READ/WRITE effect, batches only compatible phases, and commits updates solely to the owning request, thereby avoiding state corruption. On a single GPU with eight fast‑weight InPlace‑TTT streams, RW‑TTT achieves 274.61 aggregate tokens per second—9.31× faster than sequential serving and 3.44× faster than per‑stream replicas—while maintaining correct behavior on the RULER long‑context benchmark.
arXiv:2606.21803v2 Announce Type: replace Abstract: Next-token prediction is the self-supervised signal that trains language models, and every observed prompt token provides the same signal at test t...
arXiv:2608.21308v1 Announce Type: new Abstract: Test-Time Training (TTT) enables long-context processing via continuous weight updates during inference, but current methods struggle to balance the ex...
arXiv:2607. 02043v1 Announce Type: cross Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering.
arXiv:2607. 21535v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel.
arXiv:2609.17475v2 Announce Type: replace Abstract: Local agents need memory for model execution and working history. We present JustFit, an MLX runtime that coordinates their overlapping allocations...
Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap.
arXiv:2606. 19989v1 Announce Type: cross Abstract: Modern LLM training breaks a core assumption behind offline batch samplers: the true training cost of a sample is only observable after preprocessing, augmentation, templating, tokenization, and multimodal visual-token expansion.
arXiv:2607. 17652v1 Announce Type: new Abstract: Block-wise diffusion large language models (dLLMs) decode sequentially at the block level, enabling effective KV-cache reuse across blocks but making inter-block decoding strictly serial.
ReplaySSM is a technique that redefines how state-space models (SSMs) handle their recurrent state by caching SSM inputs instead of the state itself. It unrolls the recurrence into a checkpoint plus a buffer of recent inputs, reducing state memory traffic by about half and enabling efficient speculative rollback. The method also allows separate precision levels for checkpoints, improving output accuracy while accelerating decoding in hybrid models such as Mamba-2 and Gated DeltaNet.
DualPace is an LLM serving runtime that decouples scheduling between a GPU engine and an AMX‑enabled CPU engine, allowing the CPU to independently batch and advance decodes while still cooperating with the GPU for dense stages. This design expands key‑value cache capacity into host memory and reduces bursty CPU progress, leading to significant throughput gains—up to 43.2 % over NEO and 90.6 % over vLLM—without increasing latency. The system also maintains low time‑to‑first‑token and supports higher request rates at comparable tail latency.
arXiv:2606. 00516v1 Announce Type: new Abstract: Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization.
The paper presents a memory layer called galahad-kv that stores the key-value state of a large language model’s context blocks (≈16,000 tokens each) on encrypted local NVMe disk, allowing byte‑exact retrieval without recomputation. On a single NVIDIA H100, the authors tested the system with Gemma 4 12B and 31B models over 50 million tokens of real public text, achieving 100 % successful reloads, 2.8×–4.3× faster loading than recomputation, and 8.8×–12.3× lower GPU energy usage while keeping GPU memory flat. The models answered long‑range factual queries with high accuracy (82 % for 12B, 98 % for 31B) and did not hallucinate, demonstrating that stored state can extend effective context without expanding the attention window.