arXiv AI By Rajdeep Pandhare, Delzaan Bharucha

ScoutRoute: Efficient Mixture-of-Experts Routing Using Internal Language Model Signals

Read the original on arXiv AI →

ScoutRoute is a lightweight post‑hoc routing framework for Mixture‑of‑Experts systems that predicts expert competence using internal language model signals from a single forward pass of a scout model. It employs hidden‑state representations with logistic probes and logit‑derived confidence measures in a domain‑aware routing strategy, avoiding modifications to the underlying language models. Experiments on five open‑weight language models across four benchmark datasets show that ScoutRoute consistently outperforms individual routing signals and achieves superior calibration with only one scout‑model inference.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Sep 30

Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models

The paper introduces two token-error supervision methods for sparse mixture-of-experts (MoE) large language models, aligning routing affinities with token-level cross-entropy loss. The first method predicts an error score per expert and uses it to adjust affinities before top‑K selection, while the second directly aligns router affinities to the model’s objective without extra heads or inference changes. Experiments on Granite and ARC‑Challenge show accuracy gains of about 2.3–2.94 percentage points over a parameter‑matched baseline, preserving the native sparse execution budget.

By Yury Nahshan, Nati Daniel, Jacob Goldberger, Yoli Shavit