The paper introduces Personalized Federated Direct Preference Optimization (pFedDPO), which extends FedDPO by allowing each client to keep a small private LoRA adapter on top of a shared global base model. In pFedDPO, the base model is updated globally via FedAvg while the LoRA adapters remain local, capturing client‑specific preference patterns. The authors prove that this approach reduces the effective gradient diversity compared to vanilla FedDPO and demonstrate experimentally that pFedDPO achieves lower gradient norms while maintaining competitive training loss and per‑client reward accuracy.
arXiv:2601. 11219v3 Announce Type: replace-cross Abstract: Federated learning (FL) for large language models (LLMs) has attracted increasing attention as a privacy-preserving approach for adapting models over distributed data, where parameter-efficient methods such as Low-Rank Adaptation (LoRA) are widely adopted to reduce communication and memory costs.
arXiv:2608. 01556v1 Announce Type: new Abstract: Large language models are increasingly aligned to human preferences via reward modeling, but user preference data are sensitive and often cannot be centralized.
The paper introduces FedRoRA, a federated learning framework that combines Low‑Rank Adaptation (LoRA) with rank‑heterogeneous personalization. It separates model adaptation into shared global directions and client‑specific rank‑wise magnitudes, using SVD on the server to extract a global subspace and a personalized projection with top‑k selection for each client. Experiments on natural language understanding and generation tasks show that FedRoRA outperforms existing state‑of‑the‑art methods.
The paper introduces a differentially private framework for sparse reward estimation in reinforcement learning with human feedback (RLHF). It assumes that only a small subset of features influences human preferences and provides theoretical error bounds for both Sample-DP and Label-DP settings under the Bradley-Terry-Luce model. Empirical results show that the proposed methods outperform dense baselines on sentiment generation and safety alignment tasks, achieving a better privacy‑utility trade‑off.
arXiv:2606. 02563v1 Announce Type: new Abstract: Heterogeneous Differential Privacy (HDP) in Federated Learning (FL) allows clients to select individual privacy budgets ($\varepsilon_i$) according to institutional policies and data sensitivity.
Heterogeneous Differential Privacy (HDP) in Federated Learning (FL) allows clients to select individual privacy budgets ($\varepsilon_i$) according to institutional policies and data sensitivity. In practice, many HDP-FL systems employ $\varepsilon$-aware server aggregation to improve model utility by re-weighting client updates according to their declared privacy budgets.
The paper introduces a per-layer differential privacy (DP) clipping strategy for federated multilingual speech large language models (speech‑LLMs). It demonstrates that standard single‑pool per‑layer DP methods fail due to a cross‑component budget collapse caused by large norm differences between acoustic encoders and language decoders. The authors propose an α‑split two‑pool allocation that normalises encoder and decoder parameters separately, preserving the overall DP guarantee while restoring word error rate performance and providing tighter noise protection for the encoder.
arXiv:2609.36986v1 Announce Type: new Abstract: Federated LoRA fine-tuning enables parameter-efficient adaptation of pre-trained models without sharing private data, but suffers from two fundamental...
arXiv:2606. 15963v1 Announce Type: cross Abstract: Federated fine-tuning of large language models using parameter-efficient methods such as LoRA enables privacy-preserving adaptation of foundation models.
arXiv:2607. 23029v1 Announce Type: cross Abstract: Federated learning enables collaborative model training across distributed clients without centralising their data, yet privacy remains a persistent concern because the shared model updates can leak information about local datasets.
Record-level differential privacy exposes a structural misalignment in personalized federated learning when client-specific variation is low-dimensional while training repeatedly releases high-dimensi...
The paper introduces Private Best-of-N (PrivBoN), a method that adds calibrated Gumbel noise to reward scores during inference-time alignment, achieving both ε-differential privacy and KL-regularized alignment. When the privacy budget exceeds a critical threshold ε*, the noise becomes regret-optimal, matching the theoretical alignment skyline. The authors also propose Private Inference-Time Pessimism (PrivITP), which uses χ^2-regularized rejection sampling and a two-phase Gaussian mechanism to provide ex-post (ε,δ)-DP with a privacy cost independent of the number of responses, and demonstrate that both methods outperform standard Best-of-N across multiple models and datasets.