Error when attempting to run pipeline

[Justinwright973]

Hello,

I am attempting to run the Funomic pipeline, I have checked and appear to have all needed dependencies.

When I try to execute the pipeline on pre-filtered, paired end fastq data, the following happens:

FunOMIC.sh -1 out.10_S9_1.fq -2 out.10_S9_2.fq -p 10_S9.out -o testing-head-node -a /home/wright2/funOMIC/database/BacterialDB -b /home/wright2/funOMIC/database/FunOMIC-Tv1 -c "/home/wright2/funOMIC/database/FunOMIC.P.v1" -t 8

Error:

Start removing bacterial reads for 10_S9.out
[main_samview] fail to read the header from "testing-head-node/tmp/10_S9.out_Bact.sam".
Start taxonomic annotation for 10_S9.out
[E::hts_open_format] Failed to open file "testing-head-node/tmp/10_S9.out.sam" : No such file or directory
samtools view: failed to open "testing-head-node/tmp/10_S9.out.sam" for reading: No such file or directory
[E::hts_open_format] Failed to open file "testing-head-node/tmp/10_S9.out.sam" : No such file or directory
samtools view: failed to open "testing-head-node/tmp/10_S9.out.sam" for reading: No such file or directory
[E::hts_open_format] Failed to open file "testing-head-node/tmp/10_S9.out.30.sorted.bam" : No such file or directory
samtools view: failed to open "testing-head-node/tmp/10_S9.out.30.sorted.bam" for reading: No such file or directory
[E::hts_open_format] Failed to open file "testing-head-node/tmp/10_S9.out.30.c80.bam" : No such file or directory
samtools idxstats: failed to open "testing-head-node/tmp/10_S9.out.30.c80.bam": No such file or directory
Starting functional annotation for 10_S9.out
mkdir: cannot create directory 'testing-head-node/functional_profiling/tmp/': No such file or directory
/home/wright2/funOMIC/FunOMIC.sh: line 123: testing-head-node/functional_profiling/joined.fastq: No such file or directory
merged clean reads stored in: testing-head-node/functional_profiling/joined.fastq 
/home/wright2/funOMIC/FunOMIC.sh: line 127: testing-head-node/functional_profiling/log: No such file or directory
/home/wright2/funOMIC/FunOMIC.sh: line 133: testing-head-node/functional_profiling/10_S9.out.func.filtered.out: No such file or directory
awk: fatal: cannot open file `testing-head-node/functional_profiling/10_S9.out.func.out' for reading (No such file or directory)
/home/wright2/funOMIC/get_pwy_abundance_v1.py:68: DtypeWarning: Columns (3) have mixed types. Specify dtype option on import or set low_memory=False.
  jgi_ann=pd.read_csv(ann_path,delimiter='\t',names=column_names,index_col=False,dtype={'fourth_column': 'str'})
Traceback (most recent call last):
  File "/home/wright2/funOMIC/get_pwy_abundance_v1.py", line 69, in <module>
    blastx_out=pd.read_csv(sample_to_process,index_col=False,delimiter='\t',names=["qid","rid","id","lenght","mismatch","gapopne","qstart","qend","sstart","send","evalue","bscore"])
  File "/home/wright2/miniconda3/envs/funomic/lib/python3.10/site-packages/pandas/util/_decorators.py", line 211, in wrapper
    return func(*args, **kwargs)
  File "/home/wright2/miniconda3/envs/funomic/lib/python3.10/site-packages/pandas/util/_decorators.py", line 331, in wrapper
    return func(*args, **kwargs)
  File "/home/wright2/miniconda3/envs/funomic/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/wright2/miniconda3/envs/funomic/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 605, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/wright2/miniconda3/envs/funomic/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1442, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/home/wright2/miniconda3/envs/funomic/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1735, in _make_engine
    self.handles = get_handle(
  File "/home/wright2/miniconda3/envs/funomic/lib/python3.10/site-packages/pandas/io/common.py", line 856, in get_handle
    handle = open(
FileNotFoundError: [Errno 2] No such file or directory: 'testing-head-node/functional_profiling/10_S9.out.func.filtered.out'

When I check the bacterial removal log, it states the following line:

(ERR): bowtie2-align died with signal 11 (SEGV) (core dumped)

Let me know if there is anything I can clarify further, any assistance that can be provided here would be much appreciated

[Justinwright973]

Just as an update, I have consolidated that this issue occurs during the bowtie2 step only when running against the bacterialDB, but this does not occur when aligning using bowtie2 against the taxonomy DB, in summary there appears to be an issue with the bacterialDB. I redownloaded and decompressed and the issue persists!

[zixuanxie]

Hi @Justinwright973 , could you please send us your samtool version?

[promicrobial]

I'm also getting the same issue but not just with the bacterial database. The samtools version I'm running is v1.18. I can't figure out what's going wrong.

FunOMIC.sh -1 test.R1.fq -2 test.R2.fq -p test -o /home/nc/test_funomic -a /home/nc/funomic_databases/archives/BacterialDB -b /home/nc/funomic_databases/archives/FunOMIC-Tv1 -c /home/nc/funomic_databases/archives/FunOMIC.P.v1 -t 30

[vpandyarajan]

Hello, I am also trying to use this pipeline and have run into the same issue. Have there been any updates? I found that if I extracted the raw FASTA file from the database, recompiled it with bowtie then run the bowtie2 alignment I do not get such an error but I am getting unusually low amount of alignments in a sample I know has a lot of bacteria. Not sure if this helps decipher this issue at all. I may try redownloading all the raw fasta files from the uhgg and recompile the database in that way as well.

[imonteroo]

Hello. Is there any solution for this Error?

I tried to run the next loop
`
mkdir /media/microviable/SeagateExpansion/FunOMIC_output/
mkdir /media/microviable/SeagateExpansion/FunOMIC_output/Fisabio

for file in *.fastq.gz
do
base=$(basename $file .R1.fastq.gz)

printf "Analysing sample ${base} $(date)\n"

FunOMIC.sh -1 ${base%%.R1.*} \
           -2 ${base%%.R2.*} \
           -p ${base}_output \
           -o /media/microviable/SeagateExpansion/FunOMIC_output/Fisabio \
           -a /media/microviable/e/FunOMIC_databases/BacterialDB_2 \
           -b /media/microviable/e/FunOMIC_databases/FunOMIC-T \
           -c /media/microviable/e/FunOMIC_databases/FunOMIC-P \
           -t 4

now=$(date +"%T")
echo "Current time : $now"

done
I received the next screen outputAnalysing sample 51C jue 21 mar 2024 12:24:04 CET
Start removing bacterial reads for 51C_output
[E::hts_open_format] Failed to open file "/media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/tmp/51C_output_Bact.sam" : No such file or directory
samtools view: failed to open "/media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/tmp/51C_output_Bact.sam" for reading: No such file or directory
Start taxonomic annotation for 51C_output
[E::hts_open_format] Failed to open file "/media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/tmp/51C_output.sam" : No such file or directory
samtools view: failed to open "/media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/tmp/51C_output.sam" for reading: No such file or directory
[E::hts_open_format] Failed to open file "/media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/tmp/51C_output.sam" : No such file or directory
samtools view: failed to open "/media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/tmp/51C_output.sam" for reading: No such file or directory
[E::hts_open_format] Failed to open file "/media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/tmp/51C_output.30.sorted.bam" : No such file or directory
samtools view: failed to open "/media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/tmp/51C_output.30.sorted.bam" for reading: No such file or directory
[E::hts_open_format] Failed to open file "/media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/tmp/51C_output.30.c80.bam" : No such file or directory
samtools idxstats: failed to open "/media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/tmp/51C_output.30.c80.bam": No such file or directory
Starting functional annotation for 51C_output
zcat: skipping: /media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/51C_output_noBact* does not exist
merged clean reads stored in: /media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/functional_profiling/joined.fastq
awk: fatal: no se puede abrir el fichero «/media/microviable/SeagateExpansion/FunOMIC_output/Fisabio/functional_profiling/51C_output.func.out» para lectura: No existe el archivo o el directorio
/home/microviable/Programas/FunOMIC/get_pwy_abundance_v1.py:68: DtypeWarning: Columns (3) have mixed types. Specify dtype option on import or set low_memory=False.
jgi_ann=pd.read_csv(ann_path,delimiter='\t',names=column_names,index_col=False,dtype={'fourth_column': 'str'})
/home/microviable/Programas/FunOMIC/get_pwy_abundance_v1.py:70: DtypeWarning: Columns (1) have mixed types. Specify dtype option on import or set low_memory=False.
rid_to_uniprot_species=pd.read_csv(uniprot_catalog,delimiter='\t',names=["rid","uniprotId","speciesID"],dtype={'first_column': 'str', 'second_column': 'str'},index_col=False)
no protein found
`
And the bact_decontam.log file contains this text:
(ERR): "/media/microviable/e/FunOMIC_databases/BacterialDB_2/uhgg" does not exist or is not a Bowtie 2 index
Exiting now ...

[jotech]

Same problem over here. There seems to be a problem with the bacterial db (uhgg index)

bowtie2 -x ~/dat/db/funomic/BacterialDB/uhgg -1 my_R1_001.fastq.gz -2 my_R2_001.fastq.gz -S funomic_Bact.sam  
(ERR): bowtie2-align died with signal 11 (SEGV) (core dumped)

The bowtie error does only occur with the uhgg index.

I tried exactly the same version as reported in the FunOMIC publication

conda install bowtie2=2.3.4.3 samtools=1.9 diamond=2.0.8 bioconductor-keggrest flash2 pandas

Was the uhgg index updated? could you provide the original one?