Add symlink type to fromPath channel. #3791
Closed
sjpaterson
started this conversation in
Ideas
Replies: 1 comment
-
Ok, this ended up much easier to achieve than I expected.
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Request to add an option for "symbolic link" to the types in fromPath channel.
My workflow currently uses a directory full of symbolic links that point to the directories where my raw data is stored.
I then use a fromPath channel with type 'dir' which creates a process for each link that copies to scratch, does all the processing and then publishes to the same directory as the symbolic link, overwriting it with the processed data.
My desire is to be able to place new symbolic links in the directory and rerun my workflow allowing it to skip all the directories (successfully processed data) and only process the symbolic links. I currently use a script within a process to accomplish this. However, as I am passing the channel data to it, the process first copies all the data to scratch before the check can occur. As they are very large datasets this process takes a long time. Allowing the channel to only include symbolic links would prevent this.
Beta Was this translation helpful? Give feedback.
All reactions