arXiv:2607. 12696v1 Announce Type: cross Abstract: Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns.
The paper evaluates confidence‑based routing of small‑to‑large language models across multiple tasks, languages, and model families, covering 72 configurations and over 42,000 predictions. It finds that confidence routing improves performance on reading comprehension and topic classification datasets, but offers little benefit on a knowledge‑reasoning benchmark where small models remain confidently wrong. The study also shows that global confidence routing can create large language‑specific performance gaps, which equal‑quota routing mitigates with minimal overall accuracy loss.
arXiv:2602. 06154v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully.
arXiv:2606. 06098v1 Announce Type: cross Abstract: Foundational Large Language Models (LLMs) demonstrate proficiency on a wide range of general tasks, and achieve remarkable results on various specialized tasks via domain-expert LLMs.
arXiv:2512. 01113v2 Announce Type: replace-cross Abstract: Algorithmic reasoning -- the ability to perform step-by-step logical inference -- is a synthetic benchmark for evaluating multi-step reasoning abilities, designed for graph neural networks and also for transformer models.
The paper addresses the challenge of selecting demonstrations for long-context language model queries, where transformer inference costs grow quadratically with sequence length. It proposes two algorithms that distill transformer behavior into state space models (SSMs) with linear inference time, partitioning transformer layers into groups and estimating separate SSMs for each. The distilled SSMs achieve less than 0.7% approximation error, and in downstream tasks they reduce FLOPs by 14.2× while improving accuracy by 6.48% compared to baseline methods.
The paper introduces a structure‑preserving approach to prune experts in Mixture‑of‑Experts large language models, addressing memory overhead while maintaining performance. It observes that expert outputs exhibit structured redundancy, with stable rank and entropy effective rank far below the nominal number of experts. Two pruning methods—Projection Energy Expert Pruning (PEEP) and Distributional Coverage Expert Pruning (DCEP)—use router‑derived weights to preserve global structural relationships and local importance, achieving superior results on DeepSeek‑V2‑Lite and Qwen3‑30B‑A3B at 75% and 50% expert retention.
arXiv:2606. 09856v1 Announce Type: cross Abstract: Post-training Large Language Models (LLMs) for reasoning typically focuses on deductive tasks such as mathematics and coding where correctness is verifiable.
The paper introduces Ban&Pick, a post‑training, plug‑and‑play routing strategy for Sparse Mixture‑of‑Experts large language models. It identifies and reinforces a small group of highly influential experts while dynamically pruning redundant ones, leading to accuracy gains across math, code, and reasoning benchmarks. Experiments on DeepSeek and Qwen3 show notable performance improvements and a 1.25× inference speedup without retraining or architectural changes.
The paper surveys efficient reasoning in large language models, contrasting fast intuitive (System 1) and slow deep (System 2) reasoning. It analyzes why System 2 is computationally costly yet more accurate, and why System 1 is efficient but less effective. The survey covers causes of inefficiency, patterns of reasoning behavior, and potential solutions to balance performance and computational budgets, offering actionable insights and an open‑source repository for ongoing research.
arXiv:2512. 13996v3 Announce Type: replace Abstract: Sparse Mixture-of-Experts architectures are essential for scaling model capacity efficiently, yet the standard Top-$k$ routing imposes a rigid sparsity pattern that ignores the intrinsic variance in token difficulty and layer-specific computational needs.
We study the problem of demonstration selection, which involves selecting a subset of examples for prepending to a query to a language model. This problem is closely related to in-context learning and...