Leaking Validation Data in Few-Shot-Setting

[MarcoMorik]

I noticed, that the linear probing is trained with both the validation and training dataset when using weight-decay tuning.
train_loader = feature_train_val_loader

CLIP_benchmark/clip_benchmark/metrics/linear_probe.py

Line 280 in a230282

train_loader = feature_train_val_loader

This could lead to training the linear probe with more data per class when considering the few-shot setting, because the code only filters the training data [Line 235] but not the validation dataset.

So at least for the few-shot setting, it would be necessary to use only the train_loader to train the classification head of the linear probe.