arXiv Computation and Language

Perplexity Predicts Protection: Choosing Pretrained Backbones for Worst-Client Fairness in Federated Parameter-Efficient Fine-Tuning

arXiv Machine Learning

1d ago

When More Parameters Hurt: Foundation Model Priors Amplify Worst-Client Disparity Under Extreme Federated Heterogeneity

arXiv:2605.08992v2 Announce Type: replace Abstract: Federated learning (FL) is increasingly used to fine-tune foundation models (FMs) on distributed private data. The community largely assumes that l...

By Kiran Naseer, Umar Shoaib

nlp fine-tuning safety

arXiv AI

Aug 3

Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation

arXiv:2607. 28658v1 Announce Type: cross Abstract: Federated pre-training offers a way to train foundation models on private or distributed data without centralizing the underlying datasets.

By Claudia Grosser, Maike Heuer, Denis Krompass, Thomas A. Runkler

llms fine-tuning benchmarks

arXiv AI

Jun 10

From Data Heterogeneity to Convergence: A Data-Centric Review of Federated Learning

arXiv:2606. 10595v1 Announce Type: cross Abstract: Federated Learning (FL) has emerged as a promising solution for data hunger in centralized learning.

By Huong Nguyen, Micka\"el Bettinelli, Amirhossein Ghaffari, Alexandre Benoit, Hong-Tri Nguyen, Susanna Pirttikangas, Lauri Lov\'en

safety

arXiv Machine Learning

Jun 16

Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts

arXiv:2606. 15625v1 Announce Type: new Abstract: The continuous scaling of large language models (LLMs) incurs prohibitive computational costs, making Mixture-of-Experts (MoE) a scalable alternative for efficient fine-tuning via sparse activation.

By Yijun Lu, Zihan Fang, Pengpeng Qiao, Zheng Lin, Jing Yang, Yuxin Zhang, Por Lip Yee, Zhe Chen, Jun Luo

llms fine-tuning safety

arXiv AI

Sep 10

FedRAW: Preserving Rare-Label Influence in Asynchronous Federated Learning

arXiv:2609.07192v1 Announce Type: cross Abstract: Asynchronous federated learning improves scalability by updating the global model from a server-side buffer of client updates as they arrive, rather...

By Prashant Bajpai, Divya Saxena, Philippe Lalanda, German Vega

safety

arXiv Machine Learning

Aug 13

Beyond Parameter Space: NTK-Guided Personalized Aggregation for Robust Federated Learning

arXiv:2608. 12108v1 Announce Type: new Abstract: Federated learning (FL) enables collaborative model training across distributed clients while keeping data local.

By Mirko Konstantin, Stefan Zachow, Anirban Mukhopadhyay

arXiv Machine Learning

Jul 17

Supervised Fine-Tuning vs. In-Context Learning: An Equilibrium Analysis of LLM Personalization under Congestion

arXiv:2607. 14371v1 Announce Type: new Abstract: Large Language Models (LLMs) have revolutionized AI services, but a critical tension emerges: while personalization improves model performance, it consumes scarce computational resources that users must share.

By Fengzhuo Zhang, Zhuoran Yang, Dirk Bergemann

llms fine-tuning

arXiv AI

Jun 3

CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks

arXiv:2606. 03650v1 Announce Type: cross Abstract: Choosing or ranking language models for a specific application is hardest when no task-specific labeled data exists, and standard public benchmarks cannot be trusted, their items having likely leaked into pretraining, so scores reflect memorization rather than fitness.

By Alexander Apartsin, Yehudit Aperstein

llms benchmarks safety

arXiv Machine Learning

Sep 11

Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

The paper investigates how data repetition affects Mixture-of-Experts (MoE) language models compared to dense Transformers. Across models from 80 M to 1 B active parameters, MoEs degrade more quickly as data is repeated, with performance dropping significantly beyond 4× repetition and overtaking dense models only when strong regularization is applied. The study also identifies routing stabilization and expert specialization as key factors in MoE overfitting, and explores regularization techniques that can partially mitigate this issue.

By Atindra Jha, Margaret Li, Jure Leskovec, Percy Liang, Luke Zettlemoyer

llms efficiency

arXiv Machine Learning

Jul 21

PLayer-FL: A Principled Approach to Personalized Layer-wise Cross-Silo Federated Learning

arXiv:2502. 08829v2 Announce Type: replace Abstract: Federated learning (FL) with non-IID data often degrades client performance below local training baselines.

By Ahmed Elhussein, Florent Pollet, Gamze G\"ursoy

efficiency

arXiv Machine Learning

Aug 4

Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning

arXiv:2608. 01556v1 Announce Type: new Abstract: Large language models are increasingly aligned to human preferences via reward modeling, but user preference data are sensitive and often cannot be centralized.

By Seongyoon Kim, Boryeong Cho, Jihwan Oh, Seokhyun Chung, Se-Young Yun

llms reinforcement-learning fine-tuning

Hugging Face Trending Papers

Sep 7

FedRAW: Preserving Rare-Label Influence in Asynchronous Federated Learning

Asynchronous federated learning improves scalability by updating the global model from a server-side buffer of client updates as they arrive, rather than waiting for all selected clients to finish. Wh...

safety

Related stories