download script #1
Conversation
=== Do not change lines below ===
{
"chain": [],
"cmd": "poetry run python3 scripts/download_all_domains.py",
"dsid": "934d913e-8268-4342-aa0c-3702ee516d10",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
scripts/remove_downloads.py
Outdated
| from faostat_data_primap.helper.definitions import downloaded_data_path | ||
|
|
||
|
|
||
| def run(): |
There was a problem hiding this comment.
What is the delete function good for? Why would we want to delete version?
There was a problem hiding this comment.
We don't really need this script. I've used it to test the work flow. Once I have the tests set up I can delete it
scripts/remove_downloads.py
Outdated
| files_to_delete = os.listdir(path_to_files) | ||
|
|
||
| for file in files_to_delete: | ||
| path_to_file = path_to_files / file |
There was a problem hiding this comment.
Deletion of committed files only works after unlocking them using datalad unlock
|
|
||
| Returns | ||
| ------- | ||
| True if the file was downloaded, False if a cached file was found |
There was a problem hiding this comment.
In case there is no file, but download fails, requests throws and error, correct?
There was a problem hiding this comment.
Pretty sure it does. I can try and make a test for that
scripts/download_all_domains.py
Outdated
|
|
||
|
|
||
| def download_all_domains(sources: list[tuple[str]]) -> list[str]: | ||
| """ |
There was a problem hiding this comment.
You could add downloading of the metadata available with the data. Having the methodology descriptions next to the data could be quite helpful
There was a problem hiding this comment.
But It's less important and can be added later.
There was a problem hiding this comment.
I would be good to keep old version of the methodology description docs though, to see if things have changed. I assume it will not always be updated, so we might want to use checksums and only store new versions of they have changed and symlink to the old version instead. That could quickly rule out methodology changes (if they are not updated yearly)
=== Do not change lines below ===
{
"chain": [],
"cmd": "poetry run python3 scripts/download_all_domains.py",
"dsid": "934d913e-8268-4342-aa0c-3702ee516d10",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
=== Do not change lines below ===
{
"chain": [],
"cmd": "poetry run python3 scripts/download_all_domains.py",
"dsid": "934d913e-8268-4342-aa0c-3702ee516d10",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
|
@JGuetschow this is ready to be reviewed again. Since you've looked at it last time:
|
Description
Write a script that downloads all climate data from the FAOSTAT website.
We need to:
downloaded_datadirectoryChecklist
Please confirm that this pull request has done the following:
changelog/Notes
2023-11-09.gitattributesto consider.zipfiles as well