Inference efficiency

Quantization, distillation, pruning and serving work aimed at the same accuracy for less memory, latency and money.

3,248 stories · RSS feed

arXiv Machine Learning
Jul 21

DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training

arXiv:2604. 26256v2 Announce Type: replace Abstract: Reinforcement learning (RL) has become a critical paradigm for LLM post-training, yet the rollout phase -- accounting for 50--80% of total step time -- is bottlenecked by skewed generation: long-tailed trajectories indispensable for model performance block the entire training pipeline.

By Tianhao Hu, Xiangcheng Liu, Yuchun Miao, Youshao Xiao, Hongyu Zang, Yang Zheng, Xuan Huang, Jinrui Ding, Yufei Zhang, Yu Yang, Yi-Kai Zhang, Yueqing Sun, Chengcheng Han, Xiandi Ma, Wei Wang, Qi Gu, Yerui Sun, Yuchen Xie, Xunliang Cai
arXiv Machine Learning
Jul 21

NIRVANA: Structured Pruning Reimagined for Large Language Model Compression

arXiv:2509. 14230v2 Announce Type: replace Abstract: While structured pruning presents a highly effective pathway for accelerating Large Language Model (LLM) inference, existing methods frequently suffer from significant performance degradation and demand computationally retraining to recover capabilities.

By Mengting Ai, Tianxin Wei, Sirui Chen, Jingrui He
arXiv Machine Learning
Jul 21

OrderMoE: An expert similarity driven distributed edge MoE inference

arXiv:2607. 17154v1 Announce Type: cross Abstract: Although mixture-of-experts, MoE, models have been increasingly adopted to scale large language models with moderate computation cost, it remains challenging to deploy MoE inference over resource-constrained and bandwidth-limited edge infrastructures.

By Xin Yuan, Ning Li, Quan Chen, Wenchao Xu, Athanasios V. Vasilakos, Song Guo, Haijun Zhang
arXiv Machine Learning
Jul 21

Differentiable latent structure discovery for interpretable forecasting in clinical time series

arXiv:2604. 27967v2 Announce Type: replace Abstract: Background: We introduce StructGP, a continuous-time multi-task Gaussian process that couples process convolutions with differentiable structure learning to uncover a sparse, ordered directed acyclic graph (DAG) of inter-variable dependencies while preserving principled uncertainty.

By Ivan Lerner, Jean Feydy, Alexandre Kalimouttou, Anita Burgun, Francis Bach
arXiv Machine Learning
Jul 21

Deep Adaptive Bayesian Screening

arXiv:2607. 16927v1 Announce Type: cross Abstract: We introduce Deep Adaptive Bayesian Screening (DABS), a method for performing adaptive factorial screening in high-dimensional discrete design spaces.

By Jade Lejeune Herman, Arno Strouwen, Johan A. K. Suykens, Peter Goos