Setup of funannotate databases fails

[ekg123]

Hello,

I am having issue(s) with getting the funannotate databases set up. I've pasted some of the error/debugging information below. Could you offer any suggestions to get the databases installed?

I am using a funannotate v1.8.15 in a conda environment due to security restrictions our IT dept has. When I try to install the latest setting up the conda environment fails, returning the following:

conda.core.link:_execute(949): An error occurred while installing package 'bioconda::bioconductor-genomeinfodbdata

BUT, when I try the older v1.8.15 it installs properly so that I can access the environment and obtain the help screen for funannotate.

Please note that we have a helper proxy module on our HPC that allows for ncftp and wget connectivity. This proxy module is and has been active when I've been working on this installation.

With v1.8.15 installed, I added the path for FUNANNOTATE_DB to my .bashrc, and funannotate is apparently working but I cannot set up the databases with the commands:

funannotate setup -d /export/home/SomeUserWithProblems/funannotate_db #returns error 403, presumably due to a connection(ftp?) being blocked
funannotate setup -w -d /export/home/SomeUserWithProblems/funannotate_db # in an attempt to get around stock standard ftp that our HPC won't allow returns the lines below:

[Jul 25 03:01 PM]: OS: SLES 12.5, 80 cores, ~ 791 GB RAM. Python: 3.8.19
[Jul 25 03:01 PM]: Running 1.8.15
[Jul 25 03:01 PM]: Database location: /export/home/SomeUserWithProblems/funannotate_db
[Jul 25 03:01 PM]: Retrieving download links from GitHub Repo
[Jul 25 03:01 PM]: Parsing Augustus pre-trained species and porting to funannotate
Traceback (most recent call last):
File "/export/home/SomeUserWithProblems/miniforge3/envs/funannotate/bin/funannotate", line 10, in
sys.exit(main())
File "/export/home/SomeUserWithProblems/miniforge3/envs/funannotate/lib/python3.8/site-packages/funannotate/funannotate.py", line 716, in main
mod.main(arguments)
File "/export/home/SomeUserWithProblems/miniforge3/envs/funannotate/lib/python3.8/site-packages/funannotate/setupDB.py", line 730, in main
meropsDB(DatabaseInfo, args.force, args=args)
File "/export/home/SomeUserWithProblems/miniforge3/envs/funannotate/lib/python3.8/site-packages/funannotate/setupDB.py", line 175, in meropsDB
type, name, version, date, records, checksum = info.get('merops')
TypeError: cannot unpack non-iterable NoneType object

Describe the bug
With connectivity active on our HPC and the --wget flag when attempting to setup databases for funannotate, I still cannot get merops (or uniprot when I try to specify other database sources) to download and then the process falls down. Note: Augustus pre-trained models download just fine and appear in the funannotate_db directory.

What command did you issue?
funannotate setup -w -d /export/home/SomeUserWithProblems/funannotate_db

OS/Install Information
Please note I have not gotten around to installing GeneMark yet and just want to get the databases installed at this point.

Checking dependencies for 1.8.15

You are running Python v 3.8.19. Now checking python packages...
biopython: 1.83
goatools: 1.4.12
matplotlib: 3.4.3
natsort: 8.4.0
numpy: 1.24.4
pandas: 1.5.3
psutil: 6.0.0
requests: 2.32.3
scikit-learn: 1.3.2
scipy: 1.10.1
seaborn: 0.13.2
All 11 python packages installed

You are running Perl v b'5.032001'. Now checking perl modules...
Carp: 1.50
Clone: 0.46
DBD::SQLite: 1.72
DBD::mysql: 4.046
DBI: 1.643
DB_File: 1.858
Data::Dumper: 2.183
File::Basename: 2.85
File::Which: 1.24
Getopt::Long: 2.58
Hash::Merge: 0.302
JSON: 4.10
LWP::UserAgent: 6.67
Logger::Simple: 2.0
POSIX: 1.94
Parallel::ForkManager: 2.02
Pod::Usage: 1.69
Scalar::Util::Numeric: 0.40
Storable: 3.15
Text::Soundex: 3.05
Thread::Queue: 3.14
Tie::File: 1.06
URI::Escape: 5.17
YAML: 1.30
local::lib: 2.000029
threads: 2.25
threads::shared: 1.61
All 27 Perl modules installed

Checking Environmental Variables...
$FUNANNOTATE_DB=/expot/home/SomeUserWithProblems/funannotate_db
$PASAHOME=/export/home/SomeUserWithProblems/miniforge3/envs/funannotate/opt/pasa-2.5.3
$TRINITY_HOME=/export/home/SomeUserWithProblems/miniforge3/envs/funannotate/opt/trinity-2.8.5
$EVM_HOME=/export/home/SomeUserWithProblems/miniforge3/envs/funannotate/opt/evidencemodeler-1.1.1
$AUGUSTUS_CONFIG_PATH=/export/home/SomeUserWithProblems/miniforge3/envs/funannotate/config/
ERROR: GENEMARK_PATH not set. export GENEMARK_PATH=/path/to/dir

Checking external dependencies...
PASA: 2.5.3
CodingQuarry: 2.0
Trinity: Duplicate specification "KMER_SIZE=i" for option "KMER_SIZE"
augustus: 3.5.0
bamtools: bamtools 2.5.2
bedtools: bedtools v2.31.1
blat: BLAT v39x1
diamond: 2.1.8
ete3: 3.1.3
exonerate: exonerate 2.4.0
fasta: 36.3.8g
glimmerhmm: 3.0.4
gmap: 2024-05-20
hisat2: 2.2.1
hmmscan: HMMER 3.4 (Aug 2023)
hmmsearch: HMMER 3.4 (Aug 2023)
java: 22.0.1-internal
kallisto: 0.46.1
mafft: v7.526 (2024/Apr/26)
makeblastdb: makeblastdb 2.16.0+
minimap2: 2.28-r1209
pigz: 2.8
proteinortho: 6.3.2
pslCDnaFilter: no way to determine
salmon: salmon 0.14.1
samtools: samtools 1.18
snap: 2006-07-28
stringtie: 2.2.1
tRNAscan-SE: 2.0.12 (Nov 2022)
tantan: tantan 49
tbl2asn: 25.8
tblastn: tblastn 2.16.0+
trimal: trimAl v1.5.rev0 build[2024-05-27]
trimmomatic: 0.39
ERROR: emapper.py not installed
ERROR: gmes_petap.pl not installed
ERROR: signalp not installed
(funannotate) SomeUserWithProblems@athena60:12SP5:pkgs:0$ funannotate setup -w -d /export/home/SomeUserWithProblems/funannotate_db

[Jul 25 03:01 PM]: OS: SLES 12.5, 80 cores, ~ 791 GB RAM. Python: 3.8.19
[Jul 25 03:01 PM]: Running 1.8.15
[Jul 25 03:01 PM]: Database location: /export/home/SomeUserWithProblems/funannotate_db
[Jul 25 03:01 PM]: Retrieving download links from GitHub Repo
[Jul 25 03:01 PM]: Parsing Augustus pre-trained species and porting to funannotate
Traceback (most recent call last):
File "/export/home/SomeUserWithProblems/miniforge3/envs/funannotate/bin/funannotate", line 10, in
sys.exit(main())
File "/export/home/SomeUserWithProblems/miniforge3/envs/funannotate/lib/python3.8/site-packages/funannotate/funannotate.py", line 716, in main
mod.main(arguments)
File "/export/home/SomeUserWithProblems/miniforge3/envs/funannotate/lib/python3.8/site-packages/funannotate/setupDB.py", line 730, in main
meropsDB(DatabaseInfo, args.force, args=args)
File "/export/home/SomeUserWithProblems/miniforge3/envs/funannotate/lib/python3.8/site-packages/funannotate/setupDB.py", line 175, in meropsDB
type, name, version, date, records, checksum = info.get('merops')
TypeError: cannot unpack non-iterable NoneType object

[nextgenusfs]

Download urls are here: https://github.com/nextgenusfs/funannotate/blob/master/funannotate/downloads.json

See if you are able to manually download one or a few with wget

[hyphaltip]

I just upated the merops DB to 12.5 as well - can you try this again? also try the wget option if it works

[gkanogiannis]

The issue resolves when all ebi links in downloads.json are https (and not http).