Update mwa_search to current Pawsey filesystem scheme #14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* changed base data directories to be USER based, not in shared /astro/mwavcs/vcs * replaced hard-coded paths with those from config, replaced rm with munlink command * removed Galaxy config * removed duplicate parameter assignment, config overrides anyway * updated data path in docs * moved default mwa_search and vcstools versions to config file * updated default software versions for OzStar too * fix fitsdir in search flow * fixed remaining /group reference * remove explicit assignment of global config params * added software version defaults for Shanghai server
* Moved all of the params definitions into the nextflow.config and commented them * Rearranged config to load things in the right order * Replaced file inputs with path * Rewrote the beamforming so it is simplier and easier to understand * Made the config more readable and fixed a few bugs * Replaced basedir with vcsdir * Cleaned up the pulsar search module * Got the classifier working but I did install LOTAAS_wrapper.py installed in PulsarFeatureLab * Fixed up the mwa_seach_pipeline and calculated time and memory using channels correctly * Fixed the ipfb mode * Updated the --help to be more accurate * Fixed up the single pulse only search * Started making some very simple testing documentation (may be replaced with unit tests later) * Made the ddplan scripts also calculate and approximate work function * Made the pipeline split the dispersion plan into groups of equal work function size * GroupTuple the search puts by the number of dms so there are not stopping points * Made same changes to single pulse * Updated software layout to prevent instal scriptlation bugs * Started creating better documentation of our dependancies * Began updating the data_processing_pipeline.nf INCOMPLETE * Made some of the config calcs a function and made the presto version a param * Collates the prepfold jobs so they run more efficiently on HPC
…fferent colours and shapes (#10)
* Moved all of the params definitions into the nextflow.config and commented them * Rearranged config to load things in the right order * Replaced file inputs with path * Rewrote the beamforming so it is simplier and easier to understand * Made the config more readable and fixed a few bugs * Replaced basedir with vcsdir * Cleaned up the pulsar search module * Got the classifier working but I did install LOTAAS_wrapper.py installed in PulsarFeatureLab * Fixed up the mwa_seach_pipeline and calculated time and memory using channels correctly * Fixed the ipfb mode * Updated the --help to be more accurate * Fixed up the single pulse only search * Started making some very simple testing documentation (may be replaced with unit tests later) * Made the ddplan scripts also calculate and approximate work function * Made the pipeline split the dispersion plan into groups of equal work function size * GroupTuple the search puts by the number of dms so there are not stopping points * Made same changes to single pulse * Updated software layout to prevent instal scriptlation bugs * Started creating better documentation of our dependancies * Began updating the data_processing_pipeline.nf INCOMPLETE * Made some of the config calcs a function and made the presto version a param * Collates the prepfold jobs so they run more efficiently on HPC * Fixed a bug in the splice formatting for single beams * Updated find_candidate_position.nf so that it works with the new format
… will need to be updated
* Update workflow readability and data storage paths (#8) * Change data path workflow to match vcstools (#3) * changed base data directories to be USER based, not in shared /astro/mwavcs/vcs * replaced hard-coded paths with those from config, replaced rm with munlink command * removed Galaxy config * removed duplicate parameter assignment, config overrides anyway * updated data path in docs * moved default mwa_search and vcstools versions to config file * updated default software versions for OzStar too * fix fitsdir in search flow * fixed remaining /group reference * remove explicit assignment of global config params * added software version defaults for Shanghai server * Added a link to the ReadTheDocs documentation to the README * Added first pass workflow diagrams to docs folder * Added links to docs README * Attempt to add image to sphinx documentation * Update README.md * Update README.md * Cleanup Nextflow scripts to implement best practices (#7) * Moved all of the params definitions into the nextflow.config and commented them * Rearranged config to load things in the right order * Replaced file inputs with path * Rewrote the beamforming so it is simplier and easier to understand * Made the config more readable and fixed a few bugs * Replaced basedir with vcsdir * Cleaned up the pulsar search module * Got the classifier working but I did install LOTAAS_wrapper.py installed in PulsarFeatureLab * Fixed up the mwa_seach_pipeline and calculated time and memory using channels correctly * Fixed the ipfb mode * Updated the --help to be more accurate * Fixed up the single pulse only search * Started making some very simple testing documentation (may be replaced with unit tests later) * Made the ddplan scripts also calculate and approximate work function * Made the pipeline split the dispersion plan into groups of equal work function size * GroupTuple the search puts by the number of dms so there are not stopping points * Made same changes to single pulse * Updated software layout to prevent instal scriptlation bugs * Started creating better documentation of our dependancies * Began updating the data_processing_pipeline.nf INCOMPLETE * Made some of the config calcs a function and made the presto version a param * Collates the prepfold jobs so they run more efficiently on HPC * Made a few more options to give you more options to add pulsars of different colours and shapes (#10) * Find cand. position bug fix (#9) * Moved all of the params definitions into the nextflow.config and commented them * Rearranged config to load things in the right order * Replaced file inputs with path * Rewrote the beamforming so it is simplier and easier to understand * Made the config more readable and fixed a few bugs * Replaced basedir with vcsdir * Cleaned up the pulsar search module * Got the classifier working but I did install LOTAAS_wrapper.py installed in PulsarFeatureLab * Fixed up the mwa_seach_pipeline and calculated time and memory using channels correctly * Fixed the ipfb mode * Updated the --help to be more accurate * Fixed up the single pulse only search * Started making some very simple testing documentation (may be replaced with unit tests later) * Made the ddplan scripts also calculate and approximate work function * Made the pipeline split the dispersion plan into groups of equal work function size * GroupTuple the search puts by the number of dms so there are not stopping points * Made same changes to single pulse * Updated software layout to prevent instal scriptlation bugs * Started creating better documentation of our dependancies * Began updating the data_processing_pipeline.nf INCOMPLETE * Made some of the config calcs a function and made the presto version a param * Collates the prepfold jobs so they run more efficiently on HPC * Fixed a bug in the splice formatting for single beams * Updated find_candidate_position.nf so that it works with the new format --------- Co-authored-by: Sam McSweeney <[email protected]> Co-authored-by: Sam McSweeney <[email protected]> Co-authored-by: Nick Swainston <[email protected]> * add srun pre-command to all make_beam instances * resolve left-over merge conflict text --------- Co-authored-by: Sam McSweeney <[email protected]> Co-authored-by: Sam McSweeney <[email protected]> Co-authored-by: Nick Swainston <[email protected]>
bwmeyers
commented
Jan 29, 2024
|
@mengyaoxue Are you able to review this for me so we can merge? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.