Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error when blat begins #36

Open
14zac2 opened this issue Nov 25, 2020 · 5 comments
Open

Error when blat begins #36

14zac2 opened this issue Nov 25, 2020 · 5 comments

Comments

@14zac2
Copy link

14zac2 commented Nov 25, 2020

Hello!

I am experiencing an error with Flo that I was hoping you might be able to help me with. Rake aborts when blat begins, I believe. I searched for this issue and found it has happened for a few others, and noticed that one piece of advice was to update parallel. I did that through a conda environment and am now using GNU parallel 20201122, which I realize is still GNU parallel. Nonetheless, I am getting the following error:

mkdir run
cp /path/genomic.fa run/source.fa
cp /path/target.fa
faToTwoBit run/source.fa run/source.2bit
faToTwoBit run/target.fa run/target.2bit
twoBitInfo run/source.2bit stdout | sort -k2nr > run/source.sizes
twoBitInfo run/target.2bit stdout | sort -k2nr > run/target.sizes
faSplit sequence run/target.fa 20 run/chunk_
parallel --joblog run/joblog.faSplit -j 20 -a run/joblst.faSplit
29164 pieces of 29164 written
26770 pieces of 26770 written
23287 pieces of 23287 written
25387 pieces of 25387 written
25525 pieces of 25525 written
25448 pieces of 25448 written
25555 pieces of 25555 written
26474 pieces of 26474 written
25992 pieces of 25992 written
27046 pieces of 27046 written
26153 pieces of 26153 written
26728 pieces of 26728 written
26526 pieces of 26526 written
27621 pieces of 27621 written
26588 pieces of 26588 written
26266 pieces of 26266 written
26387 pieces of 26387 written
25897 pieces of 25897 written
27300 pieces of 27300 written
25692 pieces of 25692 written
parallel --joblog run/joblog.blat -j 20 -a run/joblst.blat
Loaded 2510587379 letters in 14543 sequences
Searched 116386128 bases in 23287 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 127615767 bases in 25555 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 127103007 bases in 25448 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 132315417 bases in 26474 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 132646279 bases in 26588 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 129744868 bases in 25992 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 134829202 bases in 27046 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 130548735 bases in 26153 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 126148018 bases in 25387 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 133429151 bases in 26728 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 145757029 bases in 29164 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 137556497 bases in 27621 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 131717084 bases in 26387 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 130661225 bases in 26266 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 128121430 bases in 25692 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 126631904 bases in 25525 sequences
Loaded 2510587379 letters in 14543 sequences
Searched 132127474 bases in 26770 sequences
rake aborted!
Command failed with status (3): [parallel --joblog run/joblog.blat -j 20 -a...]
/home/user/bin/flo/Rakefile:161:in parallel' /home/user/bin/flo/Rakefile:107:in block in <top (required)>'
/home/user/bin/flo/Rakefile:37:in `block in <top (required)>'
Tasks: TOP => run/liftover.chn
(See full trace by running task with --trace)

My job log looks like this:
9 : 1606275293.957 28015.902 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_11.fa rujoblog.blat
Seq Host Starttime JobRuntime Send Receive Exitval Signal Command
18 : 1606275293.989 45.953 0 0 0 9 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_06.fa run/chunk_06.fa.psl
17 : 1606275293.975 46.491 0 0 0 9 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_10.fa run/chunk_10.fa.psl
16 : 1606275293.972 872.307 0 0 0 9 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_13.fa run/chunk_13.fa.psl
5 : 1606275293.949 19148.322 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_19.fa run/chunk_19.fa.psl
12 : 1606275293.963 20725.125 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_14.fa run/chunk_14.fa.psl
7 : 1606275293.953 22003.258 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_15.fa run/chunk_15.fa.psl
2 : 1606275293.943 23220.606 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_18.fa run/chunk_18.fa.psl
11 : 1606275293.961 23502.038 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_04.fa run/chunk_04.fa.psl
8 : 1606275293.955 23821.197 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_16.fa run/chunk_16.fa.psl
3 : 1606275293.945 24157.292 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_05.fa run/chunk_05.fa.psl
13 : 1606275293.966 24351.732 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_01.fa run/chunk_01.fa.psl
1 : 1606275293.941 24630.828 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_07.fa run/chunk_07.fa.psl
14 : 1606275293.968 24792.986 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_02.fa run/chunk_02.fa.psl
4 : 1606275293.947 25343.822 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_17.fa run/chunk_17.fa.psl
6 : 1606275293.951 25997.266 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_08.fa run/chunk_08.fa.psl
19 : 1606275293.993 26497.434 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_00.fa run/chunk_00.fa.psl
15 : 1606275293.970 26656.599 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_09.fa run/chunk_09.fa.psl
20 : 1606275294.002 27909.093 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_03.fa run/chunk_03.fa.psl
9 : 1606275293.957 28015.902 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_11.fa run/chunk_11.fa.psl
10 : 1606275293.959 28659.995 0 89 0 0 blat -noHead -fastMap -tileSize=12 -minIdentity=95 run/source.fa run/chunk_12.fa run/chunk_12.fa.psl

Do you have any thoughts as to what might be going on?

Thanks so much,
Zoe
@yeban
Copy link
Collaborator

yeban commented Nov 30, 2020

I am not sure. How big is the target genome? How many sequences do you have in the target assembly? Are any of the chunks (run/chunk_xx.fa files) empty? Does it work if you set :processes: setting in flo_opts.yaml to 1?

@14zac2
Copy link
Author

14zac2 commented Dec 1, 2020

My target genome is 2.5 G and is made up of 3123 contigs. What I have done in the meantime is transported my work over to a computer cluster. Flo seems to get a little bit further but is still crashing. Right now, I am getting node failures - I thought this, at first, could have been from a faulty node in the cluster, but different nodes each crash at the same spot so I think it is something else.

What happened, was I began by running flo with a GFF file, and got an error like this one:

{earlier stuff}
Processing NW_015365746.1
Processing NW_015365747.1
Processing NW_015365748.1
Processing NW_015365749.1
Processing NW_015365750.1
Processing NW_015365751.1
mkdir run/genomic_transcripts_long
liftOver -gff /path/genomic_transcripts_long.gff run/liftover.chn run/genomic_transcripts_long/lifted.gff3 run/genomic_transcripts_long/unlifted.gff3
Reading liftover chains
Mapping coordinates
WARNING: -gff is not recommended.
Use 'ldHgGene -out=<file.gp>' and then 'liftOver -genePred <file.gp>'
/path/flo/gff_recover.rb run/genomic_transcripts_long/lifted.gff3 2> run/genomic_transcripts_long/lifted_cleanup.log | gt gff3 -tidy -sort -addids -retainids - > run/genomic_transcripts_long/lifted_cleaned.gff 2>> run/genomic_transcripts_long/lifted_cleanup.log
rake aborted!
Command failed with status (1): [/path/flo/gff_recover.rb run/GCF_001...]
/path/flo/Rakefile:60:in block (2 levels) in <top (required)>' /path/flo/Rakefile:40:in each'
/path/flo/Rakefile:40:in block in <top (required)>' /path/gems/gems/rake-13.0.1/exe/rake:27:in <top (required)>'
Tasks: TOP => default
(See full trace by running task with --trace)

I assumed this error indicated that I needed to use a gtf file instead of a gff. So I converted genomic_transcripts_long.gff to genomic_transcripts_long.gtf and am now not getting as far with flo. My system is crashing here:

{earlier stuff}
14072 pieces of 14072 written
14212 pieces of 14212 written
parallel --joblog run/joblog.blat -j 48 -a run/joblst.blat

All of my chunk_xx.fa files are populated, but my joblog.blat is empty. I am wondering if my system is running out of memory? I think next I will try to run flo on two nodes with the same number of parallel processors. Do you have any recommendations? I'm just confused as to where and why the job is failing on me.

Also, with flo failing multiple times, is there any way to place checkpoints in partial runs so that one doesn't have to start from scratch?

Thanks again,
Zoe

@14zac2
Copy link
Author

14zac2 commented Dec 2, 2020

Sorry, I realize I think I found an identical error in this other issue post: #15. I posted there, as I took the advice mentioned. Happy to then close this issue.

@14zac2 14zac2 closed this as completed Dec 2, 2020
@14zac2 14zac2 reopened this Dec 5, 2020
@14zac2
Copy link
Author

14zac2 commented Dec 5, 2020

Hi @yeban - I heeded your advice to avoid spamming another issue thread, and reopened my original one.

You mentioned that the error the multi-feature with ID "cds-XP_015332030.1" on line 301782 in file "processed.gff" has a different strand than its counterpart on line 301780 (possible in rare cases) is new. I ran gt on the original file to see if the issue was coming from it. Part of the problem was the same: there were certain mRNA features that had a Parent=gene description which, of course, pointed to a gene that didn't exist because I had filtered them out. However, once I removed the pesky Parent=gene descriptors, there were no further warnings or errors. I can run flo again on the fixed original file to see if it completes successfully.

As for why this problem occurred with my original files, I'm not sure. I was lifting over a RefSeq annotation from NCBI and filtered the gff file the following way:

gt gff3 -tidy -sort -addids -retainids original_genomic.gff > genomic_sorted_and_tidied.gff
~/flo/gff_longest_transcripts.rb genomic_sorted_and_tidied.gff > genomic_transcripts_sorted_and_tidied_long.gff

The result didn't seem to contain anything other than mRNA, CDS, or exon features, but perhaps there was some sort of bug in one of these preprocessing steps.

@yeban
Copy link
Collaborator

yeban commented Dec 11, 2020 •
edited
Loading

Hi @14zac2 - thanks for following up.

Do I understand correctly that you had to remove Parent=gene descriptor after running gff_longest_transcripts.rb script? I think this is something the script should automatically take care of, so there might be a bug to fix here.

Did you get a chance to run flo on the fixed original file to see if it produced the the multi-feature with ID "cds-XP_015332030.1" on line 301782 in file "processed.gff" has a different strand than its counterpart on line 301780 (possible in rare cases) warning again? If it does, that would suggest that lifOver is causing different exons of a transcript are mapping to different strands. I think gff_recover.rb script should then remove such annotations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yeban @14zac2