arXiv AI October 5, 2026 By Simin Fan, Alireza Abdollahpoorrostam, Martin Jaggi

OptiSelect: How does the Optimizer Shape Data Curriculum?

OptiSelect is a framework that incorporates the optimizer’s effect into online data selection for large language model pretraining. The study shows that optimizers like Lion and Muon, which use sign-based or polar-tangential preconditioners, suffer from a discriminability collapse that limits selection gains, whereas diagonal‑adaptive optimizers such as AdamW and Sophia can achieve higher gains. Experiments on 124M and 720M models confirm the theory and demonstrate that OptiSelect remains effective even when data is rephrased.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

arXiv Machine Learning

Jun 2

BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

arXiv:2510. 06048v4 Announce Type: replace Abstract: Effective data selection is essential for pretraining large language models (LLMs), enhancing efficiency and improving generalization to downstream tasks.

By Jie Hao, Rui Yu, Wei Zhang, Huixia Wang, Jie Xu, Mingrui Liu

llms benchmarks

arXiv AI

Jul 7

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

arXiv:2607. 04033v1 Announce Type: cross Abstract: Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented.

By Siyuan Li, Jiabao Pan, Yumou Liu, Zhuoli Ouyang, Xin Jin, Xinglong Xu, Jingxuan Wei, Shengye Pang, Jintao Che, Xuanhe Zhou, Conghui He, Cheng Tan

llms computer-vision benchmarks

Hugging Face Trending Papers

Sep 1

Post-Training Science for Supervised Fine-Tuning

Every supervised fine-tuning run forces the same chain of decisions, such as learning rate, batch size, LoRA or full fine-tuning, how many epochs, which optimiser, and what data to feed the model. Eac...

llms fine-tuning

arXiv Machine Learning

Sep 2

Post-Training Science for Supervised Fine-Tuning

arXiv:2609.01244v1 Announce Type: new Abstract: Every supervised fine-tuning run forces the same chain of decisions, such as learning rate, batch size, LoRA or full fine-tuning, how many epochs, whic...

By Charles O'Neill, Mudith Jayasekara, Harry Partridge

llms fine-tuning

arXiv AI

Jun 15

Beyond LoRA: Is Sparsity-Induced Adaptation Better?

arXiv:2606. 13767v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models.

By Elijah Cadenhead, Cristian McGee, Xin Li, El Houcine Bergou, Aritra Dutta

fine-tuning efficiency

arXiv Machine Learning

Aug 19

Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training

Data-DPO is a target model‑oriented supervised fine‑tuning data selection method that uses one‑step probing of the target model to generate pairwise data preferences, trains a lightweight reward model to capture these preferences, and then selects a training subset by combining target‑model preference, external quality scores, and marginal diversity. Experiments on Vision‑Flan and LLaVA‑CoT demonstrate that Data‑DPO consistently outperforms existing data selection baselines across multiple data budgets and even surpasses full data training performance.

By Peng Sun, Yi Yang, Antong Zhang, Chunxiao Li, Yanbo Wang, Dianbo Liu, xin chen, Kai Yu, Lu Chen, Tianfan Fu

llms reinforcement-learning fine-tuning

One email a morning, machine-written

Related stories

BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

Post-Training Science for Supervised Fine-Tuning

Post-Training Science for Supervised Fine-Tuning

Beyond LoRA: Is Sparsity-Induced Adaptation Better?

Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training