arXiv Machine Learning September 14, 2026 By Hyunjin Kim, Youngeun Nam, Jaemin Han, Wonhyeok Choi, Jae-Gil Lee

Clustering-Based Balanced Sampling and Allocation with Data Parallelism for High-Performance Fine-Tuning

Read the original on arXiv Machine Learning →

CluSTER is a cluster‑aware balanced sampling framework designed to improve the efficiency of instruction‑tuning for large language models. It reduces redundant computation by clustering data in gradient space and allocating samples across GPUs in a data‑parallel setting, while preserving the original distribution through weighted updates. Experiments show that CluSTER can cut training time by up to 69.6% with negligible loss in accuracy compared to existing sampling methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 31

SDO: Structure-Aware Data Organization for Efficient LLM Post-Training

arXiv:2607. 27273v1 Announce Type: new Abstract: Post-training of large language models is expensive, and existing efficiency improvements mainly focus on selecting informative samples or designing training schedules.

By Jinliang Gao, Ning Yang, Hai Wang, Baili Xiao, Pin Lyu

llms rag reinforcement-learning fine-tuning

arXiv Machine Learning

2d ago

HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training

arXiv:2609.39350v1 Announce Type: cross Abstract: As model sizes continue to scale, distributed training has become inevitable. Automatic parallelization techniques can derive efficient training para...

By Mengyuan Fan, Peizhuang Cong, Zixiao Huang, Si Xu, Tong Qiao, Yanghao Li, Jing Yang, Tong Yang, Quanlu Zhang, Yu Wang

efficiency

arXiv Machine Learning

Jun 9

Efficient Scaling of LLM Training with Flexible Context Parallelism

arXiv:2602. 21788v2 Announce Type: replace-cross Abstract: Scaling long-context capabilities is crucial for Large Language Models (LLMs).

By Yifan Niu, Han Xiao, Dongyi Liu, Wei Zhou, Jia Li

llms efficiency

arXiv AI

Jul 3

Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models

arXiv:2607. 01844v1 Announce Type: cross Abstract: This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models.

By Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao, Semih Yavuz, Silvio Savarese, Shafiq Joty

fine-tuning efficiency benchmarks

arXiv Machine Learning

Sep 25

Concurrent Split Learning Through Stable Client Clustering

The paper introduces Global Clustered Parallel Split Learning (GCPSL), a method that groups clients into fixed clusters and runs Parallel Split Learning with Global Sampling (GPSL) concurrently across these clusters, periodically merging client and server model segments. Experiments with 256 logical clients show that distributing the population across more workloads increases direct data participation, though smaller clusters may slightly reduce accuracy. In a practical four‑GPU setup, label‑aware GCPSL achieves 85 % CIFAR‑10 validation accuracy in roughly 6 minutes, compared to over 19 minutes for serialized workloads, with size‑balanced cluster assignments improving participation by 3.25 percentage points.

By Mohammad Kohankhaki, Valentin Rentschler, Anke Schmeink

Hugging Face Trending Papers

3d ago

HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training

As model sizes continue to scale, distributed training has become inevitable. Automatic parallelization techniques can derive efficient training parallelism strategies at low cost while achieving supe...

efficiency

One email a morning, machine-written

Related stories

SDO: Structure-Aware Data Organization for Efficient LLM Post-Training

HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training

Efficient Scaling of LLM Training with Flexible Context Parallelism

Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models

Concurrent Split Learning Through Stable Client Clustering

HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training