arXiv AI
ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 3
EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning
arXiv:2607. 01789v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation.
arXiv AI
Jul 22
Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
arXiv:2510. 02345v4 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) Large Language Models (LLMs) face a trilemma of load imbalance, parameter redundancy, and communication overhead.
arXiv AI
Aug 11
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
arXiv:2608. 07890v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert.
arXiv Machine Learning
Jul 7
PLoRA: Efficient Concurrent LoRA Training for Large Language Models
arXiv:2508. 02932v2 Announce Type: replace Abstract: Low-Rank Adaptation (LoRA) has gained popularity as a fine-tuning approach for Large Language Models (LLMs) due to its low resource requirements and good performance.
arXiv AI
Jul 15
CARE-LoRA: Compressed Activation REconstruction for Memory-Efficient LoRA
arXiv:2607. 11940v1 Announce Type: cross Abstract: As the scale of large pre-trained models continues to grow, fine-tuning them under limited memory budgets has become increasingly challenging.
arXiv AI
Aug 18
MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation
arXiv:2608. 15299v1 Announce Type: cross Abstract: Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heterogeneity in layer-wise redundancy.