Use of Pinned Memory for Faster CPU/GPU Bandwidth

[max-Hawkins]

I'm curious why MFC doesn't use pinned memory (see here for more info). Pinned host memory means that it will always be resident in RAM and cannot be paged out. This provides ensurances that enable faster memory transactions involving this memory region.

There is a command line flag for OpenACC to pin host memory for the entire application, and older OpenACC references mention a per-variable 'pinned' flag. In my minimal testing, I only got the first method to work. This led to increased CPU/GPU bandwidth as expected (~4GB/s to ~12GB/s for a V100 test). My thought is that this would be beneficial for I/O when we transfer to the CPU first for GPU jobs.

Change in CMakeLists.txt for the application-wide pinned memory:

PRIVATE -gpu=keep,ptxinfo,lineinfo,pinned

@henryleberre Is there some weird compiler issue I'm unaware of that would keep this from being added?