arXiv AI October 7, 2026 By Tate Berenbaum (Not Community Labs Inc.), Matias Parij (Not Community Labs Inc.), Muthaiah Venkatachalam (Intel Corporation)

Cascadia: Resident 975B MoE Inference on Eleven AI PCs

Read the original on arXiv AI →

The paper introduces Cascadia, a resident inference system that runs the 975B‑parameter Inkling mixture‑of‑experts model across eleven Intel Core Ultra X7 358H AI PCs. It details a custom MoE engine that preserves routing rules, compresses graphs for OpenVINO, and coordinates FP16 expert computation with FP32 output restoration, enabling efficient deployment of six decoder layers per machine. Performance results show up to 60.29 aggregate decode tokens per second at 88 streams, with first‑token latency of 6.05 s at fifteen streams, and demonstrate that larger context budgets recover embedded code while maintaining bounded latency.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

efficiency

arXiv AI

Aug 20

Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets

The paper presents a method for distributing large language model inference across multiple Intel AI PCs by splitting the model into pipeline shards, each pre‑compiled into an OpenVINO graph. Three key techniques—beam_idx Gather to enable GPU optimizations, speculative decoding on stateful models, and interleaved micro‑batching—allow a two‑node Llama 3.1 8B INT4 pipeline to serve two users at 1.79× the throughput of a single‑node model, while a four‑node deployment can run a 70B model that no single PC can hold. The authors provide code, benchmark logs, and reproduction scripts on GitHub.

By Tate Berenbaum, Muthaiah Venkatachalam

llms efficiency benchmarks

arXiv Machine Learning

Aug 18

Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference

arXiv:2608. 15383v1 Announce Type: new Abstract: Sparse mixture-of-experts (MoE) language models reduce arithmetic by activating only a small subset of experts per token, yet deployment still requires storing and moving the full expert bank.

By Amjad Saab

llms rag efficiency benchmarks

arXiv AI

Jun 10

Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design

arXiv:2606. 10493v1 Announce Type: cross Abstract: Local deployment of large Mixture-of-Experts (MoE) models falls short of the service quality achieved in cloud-scale environments, even under low-concurrency workloads.

By Wenxin Wang, Yule Hou, Yu Ji, Peng Qu, Youhui Zhang

efficiency

arXiv Machine Learning

Aug 11

MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs

arXiv:2510. 19366v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) scales model capacity through sparse activation, and is becoming an important architecture for large language models (LLMs).

By Xinfeng Xia, Xiaofeng Hou, Jiacheng Liu, Wenfeng Wang, Mingxuan Zhang, Peng Tang, Chao Li, Minyi Guo

llms efficiency

arXiv AI

Jul 28

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

arXiv:2607. 24434v1 Announce Type: cross Abstract: Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory.

By Dengke Han

llms efficiency

Hugging Face Trending Papers

Jul 27

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU.

llms efficiency

One email a morning, machine-written

Related stories

Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets

Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference

Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design

MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference