PubMed Download Pipeline

[star-nox]

Extract article data from XML files present in this FTP folder - https://ftp.ncbi.nlm.nih.gov/pubmed/baseline/
Store the data in Supabase
Extract remaining data using PubMed OA API
Download and store the articles in self-hosted storage

[lintrule-review]

You need to setup a payment method to use Lintrule

You can fix that by putting in a card here.

[star-nox]

Currently processing a single XML file by yielding metadata in batches of 100. It took ~37 seconds to get 100 articles + metadata ready for upload. The XML file has over 10k articles.
TO-DO:

1. Should I upload data to SQL and S3 in batches of 100 for a single file or all data per file?
2. If uploading in batches of 100, there will be conflicts in folder names - first few functions are still adding stuff to it whereas last few function will upload all articles present in that folder.