Multi-gpu training degrades performance

[XiaoXuan42]

Hi, is there any caution with this lib when use multiple gpus in training? Like taking special attention to InnerBatchnorm, etc? I train two versions of the same network structure, one on one gpu while the other on multiple gpus and their performances differ a lot.

[maxxxzdn]

Just curious, how severe is the degradation? Did you also adjust for batch size and learning rate when training on multiple GPUs?

[lrenaux-bdai]

I just posted #103 which could be what you're referring to