Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download script #1

Open
wants to merge 77 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 53 commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
9d6fe03
download script
crdanielbusch Nov 5, 2024
b38efa6
test
crdanielbusch Nov 6, 2024
b8b1c3f
[DATALAD] Download URLs
crdanielbusch Nov 7, 2024
f9ee4cf
download script from non-annex1 repo
crdanielbusch Nov 7, 2024
e4da448
Added emissions_crops
crdanielbusch Nov 7, 2024
0539921
Added emissions_crops
crdanielbusch Nov 7, 2024
b2b6d72
basic script to download files [skip ci]
crdanielbusch Nov 7, 2024
64a717b
Added farm_gate_emissions_crops
crdanielbusch Nov 7, 2024
a84c484
Added farm_gate_livestock
crdanielbusch Nov 7, 2024
9a02cc8
Added farm_gate_agriculture_energy
crdanielbusch Nov 7, 2024
55920e2
Added land_use_forests
crdanielbusch Nov 7, 2024
8e848a2
Added land_use_fires
crdanielbusch Nov 7, 2024
d234f99
Added land_use_drained_organic_soils
crdanielbusch Nov 7, 2024
efba26b
Added pre_post_agricultural_production
crdanielbusch Nov 7, 2024
e3edb3f
delete data script, unzip [skip ci]
crdanielbusch Nov 7, 2024
2fce622
remove simlinks from github [skip ci]
crdanielbusch Nov 7, 2024
f731957
Added farm_gate_emissions_crops
crdanielbusch Nov 11, 2024
d691cd8
Added farm_gate_livestock
crdanielbusch Nov 11, 2024
7e6ed5e
Added farm_gate_agriculture_energy
crdanielbusch Nov 11, 2024
897c953
Added land_use_forests
crdanielbusch Nov 11, 2024
79f0296
Added land_use_fires
crdanielbusch Nov 11, 2024
569b41d
Added land_use_drained_organic_soils
crdanielbusch Nov 11, 2024
0e4f635
Added pre_post_agricultural_production
crdanielbusch Nov 11, 2024
2b89304
scrape last updated tag from domain website
crdanielbusch Nov 11, 2024
e339125
Added farm_gate_emissions_crops
crdanielbusch Nov 11, 2024
cd66735
Added farm_gate_livestock
crdanielbusch Nov 11, 2024
74faa46
Added farm_gate_agriculture_energy
crdanielbusch Nov 11, 2024
b4bd258
Added land_use_forests
crdanielbusch Nov 11, 2024
fe3d848
Added land_use_fires
crdanielbusch Nov 11, 2024
701c69d
Added land_use_drained_organic_soils
crdanielbusch Nov 11, 2024
a18e1ad
Added pre_post_agricultural_production
crdanielbusch Nov 11, 2024
8c00042
remove zip files from github
crdanielbusch Nov 11, 2024
78d595a
git attributes to include zip files
crdanielbusch Nov 11, 2024
693566b
Added farm_gate_emissions_crops
crdanielbusch Nov 11, 2024
78eaa4a
Added farm_gate_livestock
crdanielbusch Nov 11, 2024
1b7b253
Added farm_gate_agriculture_energy
crdanielbusch Nov 11, 2024
4aeb7f2
Added land_use_forests
crdanielbusch Nov 11, 2024
fdd67fa
Added land_use_fires
crdanielbusch Nov 11, 2024
9ff12b7
Added land_use_drained_organic_soils
crdanielbusch Nov 11, 2024
82c48ee
Added pre_post_agricultural_production
crdanielbusch Nov 11, 2024
4f64464
[DATALAD RUNCMD] poetry run python3 src/faostat_data_prim...
crdanielbusch Nov 11, 2024
2ce9302
Makefile
crdanielbusch Nov 11, 2024
0a52d2e
refactor
crdanielbusch Nov 11, 2024
03af9da
refactor [skip ci]
crdanielbusch Nov 12, 2024
d7fd2af
clean up
crdanielbusch Nov 12, 2024
59ab608
[DATALAD] Recorded changes
crdanielbusch Nov 12, 2024
4d986ff
[DATALAD] Recorded changes
crdanielbusch Nov 12, 2024
ef4ced3
[DATALAD] Recorded changes
crdanielbusch Nov 12, 2024
297f191
[DATALAD] Recorded changes
crdanielbusch Nov 12, 2024
19f619d
[DATALAD] Recorded changes
crdanielbusch Nov 12, 2024
77f55d0
[DATALAD] Recorded changes
crdanielbusch Nov 12, 2024
0ee5c6a
[DATALAD RUNCMD] poetry run python3 scripts/download_all_...
crdanielbusch Nov 12, 2024
4ccafe1
clean up [skip ci]
crdanielbusch Nov 12, 2024
2033652
download methodology document [skip ci]
crdanielbusch Nov 12, 2024
d56268d
docstring [skip-ci]
crdanielbusch Nov 13, 2024
c051b34
tests [skip-ci]
crdanielbusch Nov 13, 2024
122e331
[DATALAD] Recorded changes
crdanielbusch Nov 13, 2024
6e8fae5
[DATALAD] Recorded changes
crdanielbusch Nov 13, 2024
d0986c7
[DATALAD] Recorded changes
crdanielbusch Nov 13, 2024
c3805a3
[DATALAD RUNCMD] poetry run python3 scripts/download_all_...
crdanielbusch Nov 13, 2024
5a735ba
[DATALAD] Recorded changes
crdanielbusch Nov 14, 2024
d270c58
[DATALAD RUNCMD] poetry run python3 scripts/download_all_...
crdanielbusch Nov 14, 2024
70ccf44
ci
crdanielbusch Nov 14, 2024
bfe05fa
install mypy stubs in ci
crdanielbusch Nov 14, 2024
2a99000
bs4
crdanielbusch Nov 14, 2024
32c8e6f
mypy
crdanielbusch Nov 14, 2024
bdb9b5c
mypy install missing stubs
crdanielbusch Nov 14, 2024
1926d86
install bs4 stubs directly
crdanielbusch Nov 14, 2024
9fd3087
mypy: request stubs, Path
crdanielbusch Nov 14, 2024
6fe3d39
licences, mypy
crdanielbusch Nov 14, 2024
1f3e97c
rm python datalad
crdanielbusch Nov 14, 2024
aed1cc4
poetry lock
crdanielbusch Nov 14, 2024
8560e34
deactivate poetry lock for now
crdanielbusch Nov 14, 2024
72589ed
pre commit
crdanielbusch Nov 14, 2024
4ba7f59
changelog
crdanielbusch Nov 14, 2024
8305505
integration test
crdanielbusch Nov 14, 2024
1145a23
rename
crdanielbusch Nov 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@
**/.git* annex.largefiles=nothing
* annex.largefiles=((mimeencoding=binary)and(largerthan=0))
*.csv annex.largefiles=anything
*.zip annex.largefiles=anything
5 changes: 5 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -77,3 +77,8 @@ virtual-environment: ## update virtual environment, create a new one if it does
poetry config virtualenvs.in-project true
poetry install --all-extras
poetry run pre-commit install

.PHONY: download_all_domains-environment
download_all_domains:
# downloads and stages (datalad save) all available data
datalad run poetry run python3 scripts/download_all_domains.py
Loading