arXiv Machine Learning
6d ago

Differentially Private Sparse Reward Estimation with Preference Feedback

The paper introduces a differentially private framework for sparse reward estimation in reinforcement learning with human feedback (RLHF). It assumes that only a small subset of features influences human preferences and provides theoretical error bounds for both Sample-DP and Label-DP settings under the Bradley-Terry-Luce model. Empirical results show that the proposed methods outperform dense baselines on sentiment generation and safety alignment tasks, achieving a better privacy‑utility trade‑off.

By Meng Ding, Mingxi Lei, Jie Zhang, Jinyan Liu, Di Wang
arXiv Machine Learning
Oct 5

Trade-off Functions for DP-SGD with Subsampling based on Random Allocation: Tight Upper and Lower Bounds

arXiv:2605. 06259v3 Announce Type: replace Abstract: Within the $f$-DP framework, we derive a tight analysis of the trade-off function for Differentially Private Stochastic Gradient Descent (DP-SGD) with subsampling based on random allocation in which each sample is independently assigned to exactly one of $M$ minibatches per epoch, each minibatch corresponding to one of the $M$ SGD rounds within a single epoch.

By Marten van Dijk, Murat Bilgehan Ertan