The Flow has not summarised this story yet — read it at arXiv Machine Learning.
arXiv:2608. 12385v2 Announce Type: replace Abstract: As large language models serve ever more requests, cumulative inference cost is growing relative to the one-time cost of training.
The paper introduces Cascadia, a resident inference system that runs the 975B‑parameter Inkling mixture‑of‑experts model across eleven Intel Core Ultra X7 358H AI PCs. It details a custom MoE engine that preserves routing rules, compresses graphs for OpenVINO, and coordinates FP16 expert computation with FP32 output restoration, enabling efficient deployment of six decoder layers per machine. Performance results show up to 60.29 aggregate decode tokens per second at 88 streams, with first‑token latency of 6.05 s at fifteen streams, and demonstrate that larger context budgets recover embedded code while maintaining bounded latency.
arXiv:2609.14507v1 Announce Type: cross Abstract: Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV...
arXiv:2607. 23264v1 Announce Type: cross Abstract: Fine-grained, device-initiated communication lets persistent GPU kernels in distributed diffusion transformer (DiT) inference issue remote stores and overlap data movement with Tensor Core computation.
arXiv:2606. 10493v1 Announce Type: cross Abstract: Local deployment of large Mixture-of-Experts (MoE) models falls short of the service quality achieved in cloud-scale environments, even under low-concurrency workloads.
The paper presents a method for distributing large language model inference across multiple Intel AI PCs by splitting the model into pipeline shards, each pre‑compiled into an OpenVINO graph. Three key techniques—beam_idx Gather to enable GPU optimizations, speculative decoding on stateful models, and interleaved micro‑batching—allow a two‑node Llama 3.1 8B INT4 pipeline to serve two users at 1.79× the throughput of a single‑node model, while a four‑node deployment can run a 70B model that no single PC can hold. The authors provide code, benchmark logs, and reproduction scripts on GitHub.