arXiv Machine Learning

A Deterministic Constant-Competitive Algorithm for Dynamic Mixture-of-Experts Serving

arXiv Machine Learning
4d ago

Quality-Constrained Routing over a Fixed Pool of Quantized Mixture-of-Experts Instances

The paper proposes a method for routing requests to a fixed pool of quantized Mixture-of-Experts (MoE) instances, aiming to maximize throughput while respecting a quality‑degradation budget. It introduces Fragility‑Weighted Perplexity (FWP) as a request‑specific risk metric derived from prompt tokens, and uses a window‑level linear program to compute a reduced‑reward score that aligns with the LP optimum. Experiments on Qwen prompts show that FWP‑based allocation improves throughput by 2.5% over request‑agnostic mixing and static configurations.

By Zhenghong Huang, Hongfan Wu, Jiheng Zhang
arXiv Machine Learning
Sep 10

Block-Wise Differentiable Sinkhorn Attention: Tail-Refinement Gradients with a Gap-Aware Dustbin Bridge

The paper presents a block‑wise differentiable Sinkhorn attention mechanism designed for long‑context balanced entropic optimal transport on TPU hardware. By stopping a $T$‑step Sinkhorn solve and unrolling a short refinement tail, the authors derive an exact surrogate gradient that achieves efficient block‑wise cost and memory usage. Experimental results on synthetic masked problems and a Pfam protein‑family screen demonstrate high numerical accuracy and sustained throughput on TPU v6e‑8, with notable improvements in reconstruction and sparse cross‑entropy metrics.

By Dylan Forde