arXiv:2607. 19058v1 Announce Type: cross Abstract: Optimizer state is the largest single line item in the memory budget of mixture-of-experts (MoE) training: on a 6.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
arXiv:2608. 04407v1 Announce Type: cross Abstract: Memory-efficient matrix optimizers such as Sinkhorn gradient descent remove most AdamW optimizer state for dense Transformer matrices, but direct application to Mixture-of-Experts (MoE) training is unreliable.
arXiv:2609.36222v1 Announce Type: new Abstract: Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferrin...
The paper investigates whether training Mixture-of-Experts (MoE) routers can improve memory‑bandwidth locality on consumer GPUs. Using a new zero‑surgery telemetry tool, the authors measure that a large Qwen3‑235B model is bottlenecked by disk‑based expert access, and that an LRU cache can serve a majority of requests. They pre‑register experiments training 137 M‑parameter MoE models with locality‑aware losses, finding that while cache misses can drop up to 60 % (99 % static‑pin hit rate), every configuration fails to meet a strict 1 % perplexity threshold, indicating a tight coupling between cache efficiency and model quality.
arXiv:2609.37899v1 Announce Type: new Abstract: Zero-order optimization (ZO) trains without backpropagation, making it relevant to forward-only hardware and non-differentiable loss, but its gradient...
arXiv:2608. 20210v1 Announce Type: cross Abstract: Small language models are usually built like large ones and then squeezed onto a CPU afterwards.
arXiv:2606. 07404v1 Announce Type: new Abstract: This paper reports on training a hundred-billion-parameter sparse mixture of experts on a single eight-GPU node, end to end.