The paper introduces LatentPort, a method that allows a language model to transfer its live memory to another model without requiring the receiver to reread the context. Experiments on a Qwen3.5 4B-to-9B sibling pair show that adding a Gated DeltaNet (GDN) persistent-state package reduces negative log‑likelihood by 0.747 nats/token and improves performance across 64 PG19 documents. The study also demonstrates that direct recurrent and convolution reuse outperforms learned GDN maps, and a 434,176‑parameter correction further narrows the performance gap to the native 9B model.
By Simon P. Villani
ReplaySSM is a technique that redefines how state-space models (SSMs) handle their recurrent state by caching SSM inputs instead of the state itself. It unrolls the recurrence into a checkpoint plus a buffer of recent inputs, reducing state memory traffic by about half and enabling efficient speculative rollback. The method also allows separate precision levels for checkpoints, improving output accuracy while accelerating decoding in hybrid models such as Mamba-2 and Gated DeltaNet.
By Ze-Wei Liou, Harris Nover, Amey Naik, Andrii Skliar, Igor Shovkun, Ido Shahaf, Tri Dao
RoutePack is a replay‑guided planner that coordinates expert placement and rollout‑trajectory scheduling for Mixture‑of‑Experts reinforcement learning models. By incrementally adjusting expert placement while preserving residency and then rescheduling trajectories into token‑capped micro‑batches, it jointly balances dense‑side work and layer‑wise expert peaks under fixed capacity constraints. Evaluations on 7.9B and 124B hybrid‑attention MoE models across NVIDIA H20‑3e configurations show throughput gains of 4.44‑11.85% over a DP‑local first‑fit‑decreasing baseline and 1.33‑5.28% over expert rerouting alone.
By Yibo Shen, Xudong Han, Xiaowei Zhu, Gen Li, Zhenxuan Pan
arXiv:2610.15715v1 Announce Type: new
Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...
By Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber
The paper introduces a cache‑aware post‑training framework for Mixture‑of‑Experts (MoE) models that jointly adapts the MoE backbone and lightweight auxiliary cache routers while keeping the native Top‑K expert‑selection rule. Two modes are proposed: Temporal Router, which predicts same‑layer reuse and retains experts for future tokens, and Spatio‑Temporal Router, which adds a Spatio Router that refines the temporal cache using the causal predecessor’s hidden state. Experiments on Qwen3 and GPT‑OSS across GSM8K, MATH, and CommonsenseQA show that Temporal Router improves cache hit rates and reduces expert‑weight traffic, while Spatio‑Temporal Router achieves the best load‑adjusted efficiency, outperforming strong prefetching baselines.
By Zhenhe Wu, Yaping Jin, Qinghua Xing, Hang Zhou, Wei He, Xianjie Wu, Xianfu Cheng, Jian Yang, Hanting Chen
DPA-MoE is a runtime precision control framework that allows Mixture-of-Experts (MoE) models to use any-precision routed experts. It encodes expert feed‑forward weights as prefix‑decodable chunks, profiles reconstruction residuals offline, and allocates chunks during inference based on live router gate scores. This enables a single encoded MoE checkpoint to operate at multiple effective weight budgets without retraining, improving KL divergence and downstream accuracy on Qwen3‑30B‑A3B and DeepSeekMoE‑16B with modest throughput overhead.
By Jaeyun Kim, Hoonki Lee, Susie Shin, Hyunmin Kim, Dahyun Lee, Sangyeob Kim