TurboMamba is a new language‑model architecture that combines a global Mamba‑3 model with a small Transformer to achieve faster inference, faster training, and better token prediction. It introduces three lightweight enhancements—direct access to the previous block’s last token, a small correction to next‑token scores, and a learned mixture with recent‑token frequencies—to improve local predictions. In experiments on FineWeb‑Edu, TurboMamba outperforms Transformer, Mamba‑3, Jamba, and Zamba, achieving the lowest mean test prediction loss and significantly higher training throughput and batch latency speedups.
By Hanxin Zhang
arXiv:2608. 20210v1 Announce Type: cross Abstract: Small language models are usually built like large ones and then squeezed onto a CPU afterwards.
By Christos Koutsiaris
The paper compares two low‑budget methods for converting a 30B Mixture‑of‑Experts autoregressive language model into a diffusion language model. One method updates a subset of the model’s weights in‑place, while the other freezes the context tower and conditions on a frozen causal copy via cross‑attention. With only 1B training tokens, the frozen‑tower approach achieves a HumanEval pass@10 score of 71.60 versus 6.19 for the in‑place method, and retains 95% of the parent’s GSM8K and 99% of its MMLU‑Pro performance.
By Wentao Lu, Jesse Clark, Tianyu Zhu
The paper introduces SafeStart, a technique for Mixture-of-Experts (MoE) language models that mitigates the safety degradation caused by active expert reduction during serving. By running all experts for the prompt and the first eight generated tokens before switching to a reduced set, SafeStart largely restores the model’s refusal to harmful requests while preserving most of the speedup achieved by expert reduction. Experiments on several MoE models show that SafeStart maintains high offline decode speedups (≈85% of the original) and limits the increase in harmful responses to within a few percentage points, even under jailbreak prompts.
By Erfan Baghaei Potraghloo, Saeid Shokoufa, Seyedarmin Azizi, Arya Fayyazi, Souvik Kundu, Mehdi Kamal, Massoud Pedram
Small language models are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it.
arXiv:2608.30320v1 Announce Type: new
Abstract: We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and a...
By Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, Dayiheng Liu