Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to execute hpcg with different but geometrically compatible local grid sizes #46

Open
maherou opened this issue Sep 7, 2016 · 1 comment

Comments

@maherou
Copy link
Collaborator

maherou commented Sep 7, 2016

Heterogeneous machines will need to have different local grid sizes to help balance load. We can support varying grid sizes as long as dimensions are compatible across local grid faces.

maherou added a commit that referenced this issue Sep 7, 2016
Heterogeneous machines will need to have different local grid sizes to
help balance load.  This commit includes new variables that remove the
assumption of uniform local grids across all ranks.  These new
variables (gnx, gny, gnz,gix0, giy0, giz0) are defined in
GenerateGeometry().  This commit is a partial step to addressing issue
#46.
@maherou
Copy link
Collaborator Author

maherou commented Sep 21, 2016

The first version of this feature will work for a 2-block partition of the z processor dimension. It works by specifying three new arguments to the xhpcg command:

--pz - the starting z-dimension process ID for the second block.
--zl - the value of nz in the lower block of the z processor dimension
--zu - the value of nz in the upper block

where nz is the local grid dimension in the z-dimension.

Example: If running on 1000 MPI processes, the processor grid would be 10 by 10 by 10. If you want to have different sizes for nz so that you could balance the load on a heterogeneous machine, such that nz is 64 in the lower block and 128 in the upper block, use

--pz=5 --zl=64 --zu=128

You will also need to launch the MPI program in MPMD mode passing in the compatible values for nz in both the lower and upper block. This is perhaps some tedious bookkeeping but should work.

maherou added a commit that referenced this issue Sep 21, 2016
Reference #46 for some background and usage instructions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant