clipping and noising

[npapernot]

Hello,

I'm curious about the following line:

NVFlare/nvflare/app_common/filters/svt_privacy.py

Line 119 in 0753aba

delta_w[accepted] = np.clip(delta_w[accepted] + noise, a_min=-self.gamma, a_max=self.gamma)

Could you explain why the noising is done within the clipping operation? How do you analyze the sensitivity of the algorithm given that?

Thank you!

[wyli]

The clipping was not done in the original SVT paper https://arxiv.org/abs/1603.01699, but it's used in the tumour segmentation paper https://arxiv.org/abs/1910.00962 "Gradient clipping is also applied, which is a widely-used method, acting as a model regulariser to prevent over-fitting.". In my understanding computing the sensitivity using gamma in this case becomes a more conservative (but bounded) estimation.

(converting this into a discussion)

[npapernot]

Indeed, this does not answer my initial question as I was wondering the following: if the sensitivity calculation is not relying on the clip, how is it calculated?

[wyli]

in my understanding, the sensitivity defined in this context is that given two training datasets that are neighbours (diff by one training item), the released model difference delta_w will differ at most gamma because of the clipping operation, so I think gamma is a valid upper bound for sensitivity. @npapernot would be great to have your further insight on this.

[npapernot]

But gamma is the sensitivity after clipping, not before, so I'm not sure I understand how the noise is calibrated according to the sensitivity.

[wyli]

Yes while it's helpful for stabilising delta_w for model training, I agree the current implementation makes privacy budget quantification less straightforward. cc @holgerroth Perhaps we could make this optional, providing the user choices of clipping(delta_w) + noise? (also we could have the options of clipping by value or clipping by norm. although in my experience, the model convergence can become heavily dependent on the choice of these parameters.)

[msheller]

The original SVT paper requires the input stream Q have sensitivity delta as a precondition (see the caption under Fig. 1). Answering @npapernot 's question is required by the algorithm.