data injestion. which to choose? (semi dagster related question)

[guykatz]

Hi all, new here and the data game and naturaly have many Qs...
my data source is a proprietry source. this source is a machine that holds the data in files, in binary format, that are dymanically created over time. the files are accesible via HTTPS.
At any given moment, the current inventory of data files is published via a manifest file (as a simple list of file URLs). The manifest has a well known predefined URL.
In order to retrieve data from that proprietry source, there is a need to, periodically, check the manifest for new data files, if there are new files, "take" them, transform from binary to parquet (or csv) and load into a datawarehouse (that have standard connectors).
my questions are:

1. Are there any existing connectors in dagster's echo system i can utilize that can answer my needs or that I can build some of my needs around?
2. As dagster integrates with many (Extract) libraries like meltano, airbyte, dlt, etc, how can i choose which to use? what criteria should be used? (assume a custom connector)
3. Any other suggestion i should consider? for example, the dagster tutorial of the hacker_news is somewhat similar to what I need, the top stories ID list is like my url list in the manifest and then API is used to get data per ID, similar to what i need of connecting to a URL from the manifest list... so maybe I dont need any connector 3rd party library?

On a side note, my destination warehouse seem to have a connector only in Airbyte. i am not sure if this should impact my choice of which connector library to use in writing a custom connector to my proprietry source or not...

thanks for the help!

[jfenafrique]

Check sling and dlt, then come back to check Dagster Embedded ELT