feature(streamline mutant delivery and upload)

[eliottBo]

Description

This PR is a feature branch that will implement the automation of the mutant file delivery as well as a partial rework of what files we store and upload in the FOHM upload api.

•  Create MutantSampleFileFormatter #3907
• Fetch FASTQ files from sample bundles for FOHM upload #3910
• Register new mutant delivery components #3909
• Create service to upload completed mutant analyses to caesar #3659
• Automate FOHM and GISAID uploads for mutant analyses #3887
• https://github.com/Clinical-Genomics/improve-order-flow/issues/94

Added

• MutantFileFormatter to rename correctly the files for the upload

How to prepare for test

• Ssh to relevant server (depending on type of change)
• Use stage: us
• Paxa the environment: paxa
• Install on stage (example for Hasta):

  bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-tool-stage.sh -e S_cg -t cg -b mutant-file-formatter -a

Requires

Merges of:
Clinical-Genomics/MUTANT#178
https://github.com/Clinical-Genomics/servers/pull/1520
https://github.com/Clinical-Genomics/clinical-genomics-ui/pull/620

How to test

• Test front end PR
• Update orderform
• Test mutant upload command (should deliver files to cust inbox and sync)
• Test FOHM upload
• Test delivery of other cases
• Merge Servers PR for system d unit https://github.com/Clinical-Genomics/servers/pull/1520

Expected test outcome

• Check that ...
• Take a screenshot and attach or copy/paste the output.

Review

• Tests executed by
• "Merge and deploy" approved by
  Thanks for filling in who performed the code review and the test!

This version is a

• MAJOR - when you make incompatible API changes
• MINOR - when you add functionality in a backwards compatible manner
• PATCH - when you make backwards compatible bug fixes or documentation/instructions

Implementation Plan

• Document in ...
• Deploy this branch on ...
• Inform to ...

[eliottBo]

Test in stage:
Files have the correct name after running cg deliver case -c case_internal_id -d fastq --dry-run

[islean]

Adding some comments. In general I am positive, but I do think we can make the code a bit clearer since it is taking me quite some time to puzzle together what the actual flow is here. I will try to map the flow out and you can correct me.

1. So we have SampleFiles being passed to the FileFormattingService. The first layer of formatting from the SampleFileFormatter returns a list of FormattedFiles which have the original path attached, and a formatted path which prepends the sample name?
2. These formatted files are now concatenated in the SampleFileConcatenationFormatter. This results in the formatted_path attributes becoming exchanged so that all FormattedFiles whose original_path corresponds to a file from the same sample with the same direction have the same concatenated formatted_path.
3. We now enter the MutantFileFormatter which prepends the region and region code to the file names. In doing so, all the formatted_paths from step 2 are actually set to the original paths of all the formatted files. The formatted paths are exactly the same as the original paths but with the reqion and lab code prepended.
4. We now filter out unique FormattedFiles. I guess this is because the change of the original_path in step 3 now makes it so that for a sample having 2 forward fastq files, we will have 2 duplicated FormattedFile entries? This indicates to me that we are using the list of FormattedFiles in a not intended way.
5. We now rename all the FormattedFiles from their original path to the formatted path. Will this work? I guess the original paths all correspond to the concatenated paths now so they have been created in step 2? Is the only goal at this point to rename the already concatenated fastq files so that they contain the lims metadata?

Also - in the _concatenate_fastq_files in the SampleFileConcatenationFormatter we seem to have hard coded that we are removing the raw fastq files which I thought should not happen in this case. Is it intended that we remove the raw fastq files?

[islean]

Looks good! Some unclarity stemming from Mutant with the multiple formatting steps but that is fine

[Vince-janv]

Nice work! Since we've touched the delivery flow, have you tested that non-sars-cov2 deliveries still behave as we expect them?

[diitaz93]

Awesome 🚀 Lets rename the component class if possible

[islean]

Super nice improvement!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.0% Duplication on New Code

See analysis details on SonarQube Cloud