Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Copy/Temporary context managers #118

Merged
merged 10 commits into from
Jun 19, 2024
Merged

Create Copy/Temporary context managers #118

merged 10 commits into from
Jun 19, 2024

Conversation

tjgalvin
Copy link
Owner

In an effort to nicely handle potential use of SLURM $MEMDIR and $LOCALDIR type local storage mechanisms I have attempted to spec out a couple context managers to help out.

The idea being that on entry to the context the necessarily files are copied over to the compute node local storage space while creating folders along the way, work is performed (e.g. wsclean on the local copy), and output files are copied back.

There are two managers:

  • temporarily_move_into: a subject directory or file is temporarily copied over to a temporary folder, work is performed, and it is copied back. The subject is the only item intended to be moved back, other files created along the way may not be.
  • hold_then_move_into: a folder if created where anything inside of it at the end of the context manager is copied over to the final directory.

Very much a toy at this point, but tests in tests/test_utils.py seem to mock up the expected behaviour with the use cases I have in mind. Likely can be evolved, e.g. add some extra files to place into hold_then_move_into on that folders creation.

@tjgalvin tjgalvin merged commit f95bbd9 into main Jun 19, 2024
2 checks passed
@tjgalvin tjgalvin deleted the copycontext branch June 19, 2024 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant