The paper introduces In-Context Preference-Conditioned Pareto Set Learning (ICPSL), a framework that uses a pretrained hypernetwork to generate a problem-specific, preference-conditioned Pareto solution model from limited offline data. ICPSL eliminates the need for repeated preference-wise searches or problem-specific training by producing a reusable model in a single forward pass. Experiments on synthetic and real-world problems demonstrate that ICPSL yields high-quality Pareto solutions while significantly speeding up model construction.
By Yiming Yao, Liang Zhao, Qingfu Zhang
arXiv:2606. 15115v1 Announce Type: new Abstract: Multi-objective optimization (MOO) has emerged as a powerful approach to solving complex optimization problems involving multiple objectives.
By Yiyi Zhu, Yaolin Wen, Xiang Xia, Xin An, Hanyi Si, Xiang Shu, Yangde Fu, Liang Dou, Hong Qian
arXiv:2606. 19521v1 Announce Type: new Abstract: In multi-task learning, handling an increasing number of objectives can quickly become challenging, both in terms of the computational resources and the decision maker's capacity to choose appropriate trade-offs.
By Augustina C. Amakor, Konstantin Sonntag, Sebastian Peitz
Spectral-MoE introduces a method for allocating heterogeneous per-layer expert utilization in dense-to-mixture-of-experts (MoE) conversion of large language models. It offers both static and dynamic allocators that learn to distribute a utilization budget across layers, improving performance over uniform allocation in both analytic- and learning-based conversion baselines. Experiments on Llama-2-7B, Llama-3-8B, and Qwen2.5-7B show significant reductions in perplexity and accuracy gains at various FFN utilizations.
By Zachary Broveak, Christian Kubicka, Thierry Tambe
arXiv:2605. 01712v2 Announce Type: replace Abstract: Pareto set learning (PSL) is an emerging paradigm in multi-objective optimization that trains neural networks to map preference vectors to Pareto optimal solutions.
By Xinyue Chen, Yingxuan Liang, Yiqin Huang, Chikai Shang, Hai-Lin Liu, Fangqing Gu
arXiv:2606. 01666v1 Announce Type: cross Abstract: The scaling of Large Language Models (LLMs) has driven significant performance gains but created substantial challenges in inference efficiency.
By Udbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig, Deepak Gupta
MoRA is a framework for pruning Mixture-of-Experts (MoE) models by learning a router bias for each expert and optimizing it with a language‑modeling loss and a routing‑diversity regularizer. The learned biases sharpen routing distributions to identify critical experts and encourage diverse routing preferences. After pruning, MoRA uses an expert approximation mechanism that approximates the outputs of pruned experts with affine transformations of remaining experts, further improving performance.
By Yushuai Sun, Zikun Zhou, Lin Gao, Jun Yu, Wenjie Pei
arXiv:2608. 20061v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost.
By Nayeon Kim, Hojin Lee, Yunju Bak, Jaesun Park, Boseop Kim
arXiv:2606. 29521v1 Announce Type: new Abstract: Deep learning problems rarely involve objectives that are equal in importance.
By Dara Varam, Mohamed I. Alhajri
RecSoup introduces a lightweight two‑stage method for Controllable Multi‑Objective Recommendation (CMOR) that merges multiple objective‑specific models without retraining. The approach first prunes parameters to reduce cross‑objective interference, then adaptively merges them using a learnable preference‑to‑coefficient mapping, enabling dynamic trade‑off control at inference time. Experiments show RecSoup achieves superior Pareto front quality and 165–228× faster inference than parameter‑generation baselines.
By Zidong Jia, Guofu Xie, Xiao Zhang
arXiv:2604.23036v2 Announce Type: replace-cross
Abstract: Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layer...
By Haoze He, Xingyuan Ding, Xuan Jiang, Xinkai Zou, Alex Cheng, Yibo Zhao, Juncheng Billy Li, Heather Miller
arXiv:2606. 08797v1 Announce Type: cross Abstract: Decision-focused learning has shown great promise for addressing predict-then-optimize problems, particularly in the presence of under-specified models.
By St\'ephane Eilles-Chan Way, Hugo Percot, Quentin Cappart, Tias Guns, Louis-Martin Rousseau