the result on PACS dataset is not reproduced.

[JungHunOh]

Hello,

I'm currently working on replicating the results from the PACS dataset using your code.

However, I've encountered a discrepancy as the obtained accuracies fall significantly below those reported in the paper.

For instance, when targeting the 'cartoon' domain, I achieved an accuracy of 79.78% on the PACS dataset. (reported accuracy = 82.3%)

I executed the script/train_hypo_dg.sh file.

I've not checked if there are similar issues on the other datasets.

Here is the training log: https://wandb.ai/junghunoh/hypo/reports/PACS-target-domain-cartoon---Vmlldzo3NDY5NTkw

Here are the training arguments.
{'augment': True,
'batch_size': 64,
'bottleneck': True,
'cosine': True,
'epochs': 50,
'feat_dim': 512,
'gpu': 0,
'head': 'mlp',
'id_loc': 'datasets/PACS',
'in_dataset': 'PACS',
'learning_rate': 0.0005,
'loss': 'hypo',
'lr_decay_epochs': '100,150,180',
'lr_decay_rate': 0.1,
'mode': 'online',
'model': 'resnet50',
'momentum': 0.9,
'normalize': False,
'prefetch': 4,
'print_freq': 10,
'proto_m': 0.95,
'save_epoch': 100,
'seed': 4,
'start_epoch': 0,
'target_domain': 'cartoon',
'temp': 0.1,
'trial': '0',
'use_domain': False,
'w': 2.0,
'warm': False,
'weight_decay': 0.0001}

I would appreciate if you check this issue.

Thanks.

[YixuanLi]

Thanks for flagging this issue, which we take seriously. The first authors Haoyue and Yifei will follow up.

[JungHunOh]

Thanks for your answer.
Please be aware that the training log is currently not shown since I deleted them by mistake in the wandb server.
Instead, I share the log file directly here.
output.log

[HaoyueBaiZJU]

Hello,
Thank you for your interest in our work. The script/train_hypo_dg.sh file includes default hyperparameters. However, we conduct hyperparameter tuning following common practice in DomainBed. The optimal hyperparameters vary by domain; for example, the best 'lr' for the cartoon domain is 0.0005, with 'batch_size' of 32 and 'w' of 4.0. Please refer to the appendix of our paper for a detailed range of hyperparameters. We also updated our script for the hyperparameters of each domain.