The paper investigates whether token-level routing in MoE+LoRA fine‑tuning truly isolates domain‑specific updates. Experiments with biomedical and mathematical data show that, despite near‑disjoint expert routing, adding biomedical data raises code perplexity, revealing negative transfer. Diagnostic metrics (Jaccard routing overlap and adapter‑gradient cosine similarity) pinpoint that interference stems from nearly orthogonal domain gradients competing within the same low‑rank adapter subspace. To mitigate this, the authors propose SpawnLoRA, which dynamically inserts gated sub‑adapters inside MoE experts when adapter‑level contention is detected, while keeping the router unchanged. Evaluations on Phi‑tiny‑MoE‑instruct and OLMoE‑1B‑7B demonstrate that SpawnLoRA reduces negative transfer more effectively than standard or rank‑adaptive LoRA, showing that structural separation inside experts offers benefits beyond routing or rank expansion alone.
arXiv:2609.06072v1 Announce Type: cross Abstract: Parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models commonly attaches a separate low-rank adapter to each expert. This expert-w...
arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.
arXiv:2510. 02345v4 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) Large Language Models (LLMs) face a trilemma of load imbalance, parameter redundancy, and communication overhead.
JevSoup is a training‑free framework that separates System One expert routing from System Two execution for low‑rank adaptation (LoRA) models. It selects two experts based solely on input and expert descriptions, retains the leading expert’s update, projects the second onto the orthogonal complement of the first update’s row space, and combines them with equal weights. On 14 PorTAL tasks and three Qwen3 scales, JevSoup improves task‑macro accuracy by up to 1.19 % and sample‑micro accuracy by up to 1.21 % over the strongest evaluated baselines.
The paper examines how expert pruning—removing low‑importance experts in Mixture‑of‑Experts models—fails when the router is over‑dispersed, a condition caused by aggressive load‑balancing that spreads tokens nearly uniformly across experts. In this regime, traditional importance signals from router probabilities collapse, making perplexity an unreliable predictor of downstream accuracy; for example, the lowest‑perplexity pruning on gpt‑oss‑20B harms mathematical reasoning while the highest‑perplexity pruning preserves it. To address this, the authors introduce Minimax Expert Score Allocation (MESA), a domain‑aware method that iteratively boosts scores for the most affected domain, achieving minimal worst‑case degradation across domains and outperforming baseline pruning strategies on multiple benchmarks while reducing memory usage.
ACE introduces a training‑free, calibration‑free framework for adaptive expert skipping in Mixture‑of‑Experts LLMs. It combines a Global Spectral Proxy that estimates global transformation capacity with a Router‑Conditioned Refinement that builds expert‑specific direction prototypes, enabling the model to skip low‑contribution experts while always keeping the top‑1 expert. Offline computation of expert statistics leaves only lightweight table lookups during inference, and experiments on three MoE‑based LLMs show ACE outperforms static and dynamic baselines, especially at high skipping ratios.
arXiv:2608. 07814v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) language models deliver high capacity at low per-token compute, but deploying them cheaply requires compressing their many expert weight matrices.
arXiv:2607. 01789v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation.
The paper introduces 3D-MoE, a hardware–software co‑designed system that optimizes Mixture‑of‑Experts (MoE) decoding on 3D‑stacked near‑memory processing (NMP) by exploiting expert complementarity. It groups experts offline, maps them locally to balance load, and uses a lightweight Collaborative Memory‑Network to stream hot expert weights, achieving an average 4.63× speedup over expert‑parallelism and 1.58×–1.86× throughput gains over existing NMP baselines with only a 2.11% area increase.
arXiv:2608. 10392v1 Announce Type: new Abstract: Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.
arXiv:2606. 03391v1 Announce Type: cross Abstract: Model merging has emerged as a cost-effective approach for consolidating the capabilities of multiple LLMs without retraining.
arXiv:2610.09372v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) layers replace the feed-forward block of a Transformer with E expert networks, and each token is routed to k of these experts....