Maximum size of FASTA file used

[SC0O8Y]

Hello :-)

I am trying to run a meta-proteomics analysis using 1.9.1 and i am using a very large fasta file. I notice that it took about 30 hours to load the file into ram ~600GB, but it crashed in the end. I would love to run a DB-reducer but I also cannot seem to get them to work.

Is there possibly something i am doing wrong?

Also a side note: MSFileReader found: MSFileReader Core 0 (ignored, version required: 31, Thermo .raw data processing unavailable)
I have 3.1 installed at SP4 I believe, I am not sure why it is still not being recognised, any ideas I would appreciate immensely

Windows data server 2023, AMD Epyc 256 cores, 1.8TB ram, 2x 6TB NVME.2 in raid0

Kind regards,
Joel

[jjGG]

Hei Joel,
How large is your file? Did you turn of Mass-calibration and use split-DB?
cheeers - jonas

[vdemichev]

Hi Joel,

Can you please share the log?
DIA-NN needs a specific MSFileReader version (available by the link here: https://github.com/vdemichev/DiaNN?tab=readme-ov-file#raw-data-formats), i.e. SP4 will not work.

In general, DIA-NN 1.9.2 (release soon) will have a signficant (potentially 2x) reduction of RAM usage for storing the library, which can be helpfulf for huge databases. Also, searching the database in parts can help (i.e. if you have 500 species, search the first 50, then the second 50, etc).

Best,
Vadim

[SC0O8Y]

Hi Vadim (And Jonas :-) )

(Firstly, I want to say a huge thank you to in general, Im a tad bit of a fangirl)

Thank you regarding the file reader, I had this issue before and when creating the new VM forgot about this verison of Msfilereader.

I will try and get the log but the VM it was on had a tad bit of a hiccup at that location and I may need to roll back the drive image.

The file is 120GB, (Uniprot, SwissProt and trEMBL) I left on mass calibration, and I also didnt know split db was an option within straight DIA-NN, I must have missed the line for it?
is it like --split-db# or something akin to this? I tried utilising iMetaLab DIA to use DB-Reducer and was running into a lot of java based issues, Fragpipe did not like the database size either but maybe I need to manually create decoys in R

The project consists of 7TB of metaproteomics data ran on the Astral on 40m methods.
For QC using an older library that we constructed utilsing DDA based human fecal samples we achieved 30,000 proteins from a 800,000 protein library identified.
We dont have a full assembly of the transcriptome yet, so using a wider database and the IGC+ or human gut database should complement and round out the identification.

Thank you both for your help, i will go hunting for the log :-)

Any advice i would appreciate immensly

[jjGG]

@SC0O8Y, sorry about the confusion on my side with split-db and mass calibration. These are important for FragPipe-MSFragger when using MetaproteomicsDBs.

[vdemichev]

Btw, MSFragger-DIA is always an option, can use it to create a DIA-based lib and then feed this lib into DIA-NN. So if MSFragger can handle this FASTA, then all is good :)

[vdemichev]

OK, no chance it will be able to process 120Gb FASTA, even with 600 GB RAM, just not gonna fit.
I would just suggest to do some preliminary analyses with a lower number of species, i.e. not pushing it this high, to see if you get significant advantage over using a DDA-based lib.

Best,
Vadim

[SC0O8Y]

Hi Vadim. I have 1.8TB of ram on this computer, would that work? Kindest regards Joel

…

On Wed, 18 Sept 2024, 8:05 pm Vadim Demichev, ***@***.***> wrote: OK, no chance it will be able to process 120Gb FASTA, even with 600 GB RAM, just not gonna fit. I would just suggest to do some preliminary analyses with a lower number of species, i.e. not pushing it this high, to see if you get significant advantage over using a DDA-based lib. Best, Vadim — Reply to this email directly, view it on GitHub <#1171 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG7EVUH5YYJFAMNSK4BPP4LZXFF7RAVCNFSM6AAAAABOIRGOXCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANRXHE4TIMY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[vdemichev]

Hi Joel,

With 120 Gb FASTA - no. It's not a typo, i.e. FASTA is 120 Gb not Mb? Because a regular human FASTA is like 50Mb for 70k sequences, so here you have > 100 million sequences in the FASTA?

Best,
Vadim

[SC0O8Y]

Ahhh, that's not a bad idea, I was having issues with a time out setting. But nearly achieved it. I will give it another try

…

On Wed, 18 Sept 2024, 9:53 pm Vadim Demichev, ***@***.***> wrote: Btw, MSFragger-DIA is always an option, can use it to create a DIA-based lib and then feed this lib into DIA-NN. So if MSFragger can handle this FASTA, then all is good :) — Reply to this email directly, view it on GitHub <#1171 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG7EVUF5VAG6R6QY7LHUXYTZXFSVHAVCNFSM6AAAAABOIRGOXCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANRYGA4DANY> . You are receiving this because you were mentioned.Message ID: ***@***.***>