← Back to all news
arXiv AI By Mina Mirzadehsarcheshmeh, Amir Keyvan Khandani

PAC-Private Autoregressive Generation: Calibrating Noise to Ensemble Disagreement

Read the original on arXiv AI →

The paper introduces PAC‑Private Autoregressive Generation, a method that calibrates noise based on ensemble disagreement across overlapping ‘worlds’ of a private corpus, thereby extending PAC privacy from classification to text generation. By training adapters on a frozen public model and using posterior‑weighted disagreement to add noise only when predictions vary, the approach achieves strong privacy guarantees while preserving most of the fine‑tuning benefit. Experiments on WikiText‑103 with GPT‑2‑small show 74 % of the fine‑tuning gain retained with a per‑token budget of 2⁻³², and membership‑inference success bounded to 51.08 % after one million tokens, outperforming PMixED under matched conditions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Aug 28

Privacy Without Regret: Differentially Private Inference-Time Alignment

The paper introduces Private Best-of-N (PrivBoN), a method that adds calibrated Gumbel noise to reward scores during inference-time alignment, achieving both ε-differential privacy and KL-regularized alignment. When the privacy budget exceeds a critical threshold ε*, the noise becomes regret-optimal, matching the theoretical alignment skyline. The authors also propose Private Inference-Time Pessimism (PrivITP), which uses χ^2-regularized rejection sampling and a two-phase Gaussian mechanism to provide ex-post (ε,δ)-DP with a privacy cost independent of the number of responses, and demonstrate that both methods outperform standard Best-of-N across multiple models and datasets.

By Ishi Jain, Nandini Bhattad, Sayak Ray Chowdhury
llmsreinforcement-learningsafety
More like this →
arXiv Machine Learning
Aug 20

Topology-Aware Differential Privacy in Hierarchical Federated Learning

The paper introduces Fulcrum, a topology‑aware differential privacy scheme for hierarchical federated learning that allocates noise based on the size and exposure of regional aggregation groups. By deriving a closed‑form exposure dispersion metric from region structure and weights, the method optimally balances privacy and utility, achieving up to 14.84% accuracy gains on image tasks and 12.16% on text tasks at ε = 0.99 compared to uniform noise allocation. The approach ensures each participant receives noise commensurate with its actual exposure, eliminating unnecessary privacy overhead.

By Murtaza Rangwala, Richard O. Sinnott, Rajkumar Buyya
nlpsafety
More like this →