Sbatch fails due to parameters not being provided (though they are)

[ycsong]

Hello,

Executing the following code to run metabat:

bash metaGEM.sh -t metabat -j 1 -c 24 -m 80 -h 10

I noticed the following in the nohub.out:

sbatch: error: You must specify a the number of nodes to use
sbatch: error: Batch job submission failed: Job size specification needs to be provided
Error submitting jobscript (exit code 1):

Job failed, going on with independent jobs.
Exiting because a job execution failed. Look above for error message
Complete log: /home/bioinfo_tools/bin/metaGEM/workflow/.snakemake/log/2024-01-10T150739.993699.snakemake.log
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cluster nodes: 1
Conda environments: ignored
Job counts:
	count	jobs
	1	all
	1
Select jobs to execute...

[Wed Jan 10 15:12:37 2024]
Job 0: 
        WARNING: Be very careful when adding/removing any lines above this message.
        The metaGEM.sh parser is presently hardcoded to edit line 22 of this Snakefile to expand target rules accordingly,
        therefore adding/removing any lines before this message will likely result in parser malfunction.

I find this to be strange, as I've already specified this in the parameter. Also, does metaGEM submits the metabat job to the server using sbatch?

Something I also noticed after executing step is that the jobs don't appear to have been submitted, as when I check with showq or uqueue command, the jobs aren't there.

Your help would be appreciated as always. Thank you.

[franciscozorrilla]

Hey Young,
Yes, the sbatch command and snakemake are used to submit jobs to the slurm scheduler, for example:

metaGEM/workflow/metaGEM.sh

Line 427 in 57cfd60

[Yy]* ) echo "nohup snakemake all -j $njobs -k --cluster-config ../config/cluster_config.json -c 'sbatch -A {cluster.account} -t {cluster.time} -n {cluster.n} --ntasks {cluster.tasks} --cpus-per-task {cluster.n} --output {cluster.output}' &"|bash; break;;

Could you check what version of snakemake you are using? It should be >=5.10.0,<5.31.1.
Also, could you show me what your config files look like? It looks like probably your wildcards are not expanding properly based on the dataset folder.

[ycsong]

Hi Francisco,

Apologies for the delayed response. Our server was shut down due to a maintenance yesterday.

The version of snakemake I have at the moment is 5.31.0

Here is my config.yaml:

path:
    root: /home/bioinfo_tools/bin/metaGEM
    scratch: /home/bioinfo_tools/bin/metaGEM/scratch
folder:
    data: dataset
    logs: logs
    assemblies: assemblies
    scripts: scripts
    crossMap: crossMap 
    concoct: concoct
    maxbin: maxbin
    metabat: metabat
    refined: refined_bins
    reassembled: reassembled_bins
    classification: GTDBTk
    abundance: abundance
    GRiD: GRiD
    GEMs: GEMs
    SMETANA: SMETANA
    memote: memote
    qfiltered: qfiltered
    stats: stats
    proteinBins: protein_bins
    dnaBins: dna_bins
    pangenome: pangenome
    kallisto: kallisto
    kallistoIndex: kallistoIndex
    benchmarks: benchmarks
    prodigal: prodigal
    blastp: blastp
    blastp_db: blastp_db
scripts:
    kallisto2concoct: kallisto2concoct.py
    prepRoary: prepareRoaryInput.R
    binFilter: binFilter.py
    qfilterVis: qfilterVis.R
    assemblyVis: assemblyVis.R
    binningVis: binningVis.R
    modelVis: modelVis.R
    compositionVis: compositionVis.R
    taxonomyVis: taxonomyVis.R
    carveme: media_db.tsv
    toy: download_toydata.txt
    GTDBtkVis: 
cores:
    fastp: 4
    megahit: 48
    crossMap: 48
    concoct: 48
    metabat: 48
    maxbin: 48
    refine: 48
    reassemble: 48
    classify: 2
    gtdbtk: 48
    abundance: 16
    carveme: 4
    smetana: 12
    memote: 4
    grid: 24
    prokka: 2
    roary: 12
    diamond: 12
params:
    cutfasta: 10000
    assemblyPreset: meta-sensitive
    assemblyMin: 1000
    concoct: 800
    metabatMin: 50000
    seed: 420
    minBin: 1500
    refineMem: 1600
    refineComp: 50
    refineCont: 10
    reassembleMem: 1600
    reassembleComp: 50
    reassembleCont: 10
    carveMedia: M8
    smetanaMedia: M1,M2,M3,M4,M5,M7,M8,M9,M10,M11,M13,M14,M15A,M15B,M16
    smetanaSolver: CPLEX
    roaryI: 90
    roaryCD: 90
envs:
    metagem: envs/metagem
    metawrap: envs/metawrap
    prokkaroary: envs/prokkaroary

[franciscozorrilla]

thanks, and do you have samples in subfolders within your dataset/ folder?

[ycsong]

Hi Francisco,

Yes, I have the fastq and the contig file in a subfolder within dataset/ folder.

Within the /home/bioinfo_tools/bin/metaGEM, I have the following folders:
-> config
-> dataset
-> scratch
-> workflow

[franciscozorrilla]

Hey Young, sorry for the delayed response, I was on vacation.
I see, thank you for confirming. I think the dataset folder should be inside the workflow folder, which is probably why your wildcards are not expanding. Could you move the folder and try to submit jobs again?

[ycsong]

Hi Francisco,

Thank you for reaching out, and no worries about the delay. It's actually a good timing, given that our server here has suffered some technical issues, and we are waiting to have it fixed.

I'll definitely give this a go and will post updates as soon as they become available.

Thanks for your help so far, it is much appreciated.

[ycsong]

Hi Francisco,

I just tried as you have previously recommended. The issue that I described seems to have been resolved, except I am getting different issues:

snakemake all -j 1 -n -k --cluster-config ../config/cluster_config.json -c "sbatch -A {cluster.account}"
Building DAG of jobs...
MissingInputException in line 390 of /home/bioinfo_tools/bin/metaGEM/workflow/Snakefile:
Missing input files for rule crossMapSeries:
/home/bioinfo_tools/bin/metaGEM/qfiltered

In my dataset folder, I have a subfolder that contains the contigs and the reads files. The reads files are both in .fastq.gz format, while the contig file is a nucleotide fasta file (.fna). Should these files be in the dataset folder?

Thank you,

Young

[franciscozorrilla]

Hi Young,

Great that we are making some progress!

Just FYI, metaGEM looks at the subfolders in the /dataset folder in order to expand the wildcards used by Snakemake. metaGEM expects that you will start from the raw fastq files which are taken from each aforementioned subfolder, quality filters them, and stores them in subfolders under /qfiltered. If you have already generated any results without metaGEM, you just need to name and store them as expected by the corresponding Snakefile rule (example below). Next, metaGEM looks for files in the /qfiltered folder in order to assemble them and deposit the contigs in subfolders under /assemblies, and so on for the rest of the pipeline.

You get the idea: the output of one rule is used as the input for another. If Snakemake sees that an input is missing, it looks for the rule that generates that input file. In your case you are trying to run metabat, which requires a file summarizing the mapping results from the quality filtered reads to your assemblies. Since you are missing this required file, then the rule to generate it is
brought up (crossMapSeries):

metaGEM/workflow/Snakefile

Lines 390 to 393 in 57cfd60

	rule crossMapSeries:
	input:
	contigs = rules.megahit.output,
	reads = f'{config["path"]["root"]}/{config["folder"]["qfiltered"]}'

However that rule is unable to find/generate the required input (i.e. qfiltered reads) as we saw in your error message

Missing input files for rule crossMapSeries:
/home/bioinfo_tools/bin/metaGEM/qfiltered

Have a look at the qfilter rule to see the expected inputs/outputs:

metaGEM/workflow/Snakefile

Lines 145 to 151 in 57cfd60

	rule qfilter:
	input:
	R1 = DATA_READS_1,
	R2 = DATA_READS_2
	output:
	R1 = f'{config["path"]["root"]}/{config["folder"]["qfiltered"]}/{{IDs}}/{{IDs}}_R1.fastq.gz',
	R2 = f'{config["path"]["root"]}/{config["folder"]["qfiltered"]}/{{IDs}}/{{IDs}}_R2.fastq.gz'

Where the reads are defined as:

metaGEM/workflow/Snakefile

Lines 16 to 17 in 57cfd60

	DATA_READS_1 = f'{config["path"]["root"]}/{config["folder"]["data"]}/{{IDs}}/{{IDs}}_R1.fastq.gz'
	DATA_READS_2 = f'{config["path"]["root"]}/{config["folder"]["data"]}/{{IDs}}/{{IDs}}_R2.fastq.gz'

As detailed in the quickstart quide, metaGEM expects the fastq file to be named after the sample ID, delimited by the underscore _, marking the forward/reverse reads R1 or R2, and with the extension .fastq.gz. These are the conventions we use, but you can easily modify the Snakefile to take in samples with any naming scheme you may want. Fixing your filenames and ensuring they are in the correct subfolders as expected by the Snakefile rules you are trying to use should resolve your issues.

Good luck!
Francisco

[ycsong]

Hi Francisco,

I think we are nearly there. I received a sbatch error as show below:

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cluster nodes: 1
Conda environments: ignored
Job counts:
        count   jobs
        1       all
        1       crossMapSeries
        1       megahit
        1       metabatCross
        4
Select jobs to execute...

[Fri Jan 26 15:22:22 2024]
rule megahit:
    input: /home/bioinfo_tools/bin/metaGEM/qfiltered/CFS1-BTM/CFS1-BTM_R1.fastq.gz, /home/bioinfo_tools/bin/metaGEM/qfiltered/CFS1-BTM/CFS1-BTM_R2.fastq.gz
    output: /thome/bioinfo_tools/bin/metaGEM/assemblies/CFS1-BTM/contigs.fasta.gz
    jobid: 2
    benchmark: /home/bin/metaGEM/benchmarks/CFS1-BTM.megahit.benchmark.txt
    wildcards: IDs=CFS1-BTM

sbatch: error: You must specify a the number of nodes to use
sbatch: error: Batch job submission failed: Job size specification needs to be provided

It appears that the input files are now being read correctly.
I think I already specified the number or nodes in the cluster_config.json (under the field, "n") :

{
"__default__" : {
        "account" : "account_id",
        "time" : "0-10:00:00",
        "n" : 1,
        "tasks" : 1,
        "mem" : 80G,
        "name"      : "DL.{rule}",
        "output"    : "logs/{wildcards}.%N.{rule}.out.log",
},
}

Right now this is the command I am executing:

bash metaGEM.sh -t metabat -j 1 -c 1 -m 80 -h 10

But I saw it in the nohup message that there are total of four jobs. So does the '-j 1' need to be changed to '-j 4'?

[ycsong]

Hi Francisco,

I think I may have found a source to the sbatch issue:

snakemake all -j $njobs -n -k --cluster-config ../config/cluster_config.json -c "sbatch -A {cluster.account} -t {cluster.time} -n {cluster.n} --ntasks {cluster.tasks} --cpus-per-task {cluster.n} --output {cluster.output}"

Is cluster.n supposed to be the number of nodes? If so, I think the -n should be -N or --nodes (in sbatch -n is number of tasks, which is already stated by using --ntasks)

There were other similar lines there, so I replaced these instances and then executed the codes. The job submission was successful (though execution was not, which I will address as a new discussion topic).

[franciscozorrilla]

Please read the slurm documentation to better understand what each flag means. Of course you can modify the sbatch call to suit your needs, but the way that the sbatch call is configured is to submit -j jobs at a time with -c (metaGEM.sh flag) / -n (cluster_config.json flag) cores each. You should adjust the requested cores based on your HPC infrastructure and the tasks you are trying to accomplish (e.g. assembly will required more cores than quality filtering)