DP-SGD
Clip per-example gradients + add Gaussian noise. Aggregate over batch. Standard for private ML.
Advertisement
ε-DP
Privacy budget ε. Small ε = strong privacy. Trade-off: small ε → high noise → lower model utility.
Advertisement
For LLM pretraining
ε < 10 achievable on LLM pretraining. Modest utility hit. Deployed at Apple, Google.
For fine-tuning
Tighter ε feasible. Practical for enterprise deploying on sensitive data.