Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Staging area for the census tract data (similarity) #130

Merged
merged 122 commits into from
Nov 15, 2024
Merged

Staging area for the census tract data (similarity) #130

Changes from all commits
Commits
Show all changes
122 commits
Select commit Hold shift + click to select a range
268ee04
created sql db
rfl-urbaniak Mar 6, 2024
7df6f9e
started sql migration
rfl-urbaniak Mar 6, 2024
c84a537
conversion of csvs to db
rfl-urbaniak Mar 6, 2024
50e93ba
small speed test
rfl-urbaniak Mar 8, 2024
02dd6ce
data cleaning scripts migrated to a subfolder
rfl-urbaniak Mar 9, 2024
d8cced8
fixed pytest version at 7.4.3
rfl-urbaniak Mar 9, 2024
c127307
data export from csv to db with test
rfl-urbaniak Mar 9, 2024
9c8d80c
fix indentation dg
rfl-urbaniak Mar 10, 2024
d2cb7d2
WIP
rfl-urbaniak Mar 10, 2024
c2b731e
DataGrabberDB with tests
rfl-urbaniak Mar 10, 2024
e5f8a4b
refactored DataGrabberCSV
rfl-urbaniak Mar 10, 2024
5f942cf
passed DataGrabberDB downstream
rfl-urbaniak Mar 10, 2024
b46dc5a
performance tests
rfl-urbaniak Mar 12, 2024
648974e
removed vscode settings
rfl-urbaniak Mar 12, 2024
7c43efa
lint with new mypy and pyro, ignore Adam mypy complaint
rfl-urbaniak Mar 12, 2024
79f74d7
added staging-* to workflow
rfl-urbaniak Mar 12, 2024
1312ff6
force the most recent version of isort in setup
rfl-urbaniak Mar 12, 2024
2c5a8b1
Merge branch 'staging-county-data' of https://github.com/BasisResearc…
rfl-urbaniak Mar 12, 2024
e5919f1
typo in isort import
rfl-urbaniak Mar 12, 2024
b3664ce
isort modeling_utils.py
rfl-urbaniak Mar 12, 2024
7f47359
switch to --apply within isort in clean.sh
rfl-urbaniak Mar 12, 2024
41cda1f
removed --apply as redundant
rfl-urbaniak Mar 12, 2024
839a4ab
upgrade black
rfl-urbaniak Mar 12, 2024
88bc3ef
add black profile to scripts
rfl-urbaniak Mar 12, 2024
99ddeb0
removed black from nbqa
rfl-urbaniak Mar 12, 2024
0a37867
dealing with linter versions
rfl-urbaniak Mar 12, 2024
475c732
revising workflow
rfl-urbaniak Mar 12, 2024
a9bc78d
db pipeline to workflow
rfl-urbaniak Mar 12, 2024
d5062a1
suspend black to avoid linting version issues
rfl-urbaniak Mar 12, 2024
4764f3d
Merge branch 'staging-county-data' of https://github.com/BasisResearc…
rfl-urbaniak Mar 12, 2024
a0f97a5
decouple db pipeline from data grabber
rfl-urbaniak Mar 12, 2024
c13a431
lint
rfl-urbaniak Mar 12, 2024
1885390
runner isort recommendations by hand
rfl-urbaniak Mar 12, 2024
8e23fd3
suspend isort switch to black
rfl-urbaniak Mar 12, 2024
f7fb0ec
switch to dev install (as torch is required to test inference now)
rfl-urbaniak Mar 12, 2024
cfde8bc
suspend notebook tests
rfl-urbaniak Mar 12, 2024
5ca57e1
typo in test yaml
rfl-urbaniak Mar 12, 2024
be24f85
remove parallel testing as different tests are collected at different…
rfl-urbaniak Mar 12, 2024
71bc2b9
fixing test.yml
rfl-urbaniak Mar 12, 2024
a0c0526
fixed pyro version to 1.8.5
rfl-urbaniak Mar 12, 2024
9fe9130
removed redundant code from test_inference
rfl-urbaniak Mar 13, 2024
74cb94d
format lint
rfl-urbaniak Mar 13, 2024
9b79af0
working on population CT WIP
Niklewa Jun 20, 2024
5a4d00d
working on the cleaning script WIP
Niklewa Jun 21, 2024
a40ebc1
adjusting VariableCleaner class, adding processed population datasets…
Niklewa Jun 24, 2024
18140f6
adding DG class for census tracts
Niklewa Jun 25, 2024
29f5f0d
creating DG tests for cenus_tract level
Niklewa Jun 25, 2024
578ab87
lint and format
Niklewa Jun 25, 2024
c0b8762
starting cleaning ethnicity for CT WIP
Niklewa Jun 25, 2024
beddecf
lint
Niklewa Jun 25, 2024
f325df6
adding cleaning process for ethnic_composition data WIP
Niklewa Jun 27, 2024
af986a9
refactoring data grabber
Niklewa Jun 28, 2024
69a6f86
lint
Niklewa Jun 28, 2024
c81a604
starting urbanicity
Niklewa Jun 28, 2024
672ca48
retrieving urbanicity WIP
Niklewa Jul 1, 2024
c527f4b
updated scripts
rfl-urbaniak Jul 1, 2024
9dd2e41
format, lint
rfl-urbaniak Jul 1, 2024
c3743d2
some changes regarding the review
Niklewa Jul 2, 2024
15f0b0e
adding scraping folder and script for population
Niklewa Jul 3, 2024
c179525
working on urbanicity WIP
Niklewa Jul 3, 2024
85dfa84
added types requests to setup
rfl-urbaniak Jul 3, 2024
8d96737
types request to tests
rfl-urbaniak Jul 3, 2024
520a776
changing a clean_variable argument name
Niklewa Jul 4, 2024
30f06ed
working on urbanicity
Niklewa Jul 4, 2024
a514338
lint
Niklewa Jul 4, 2024
50aeb8c
finished raw data loading
Niklewa Jul 4, 2024
3886e77
Merge pull request #124 from BasisResearch/add-population-census-tracts
rfl-urbaniak Jul 4, 2024
5c22cce
starting income WIP
Niklewa Jul 4, 2024
d786f5b
Merge branch 'staging-census-tracts' of https://github.com/BasisResea…
Niklewa Jul 5, 2024
a9370c8
adding raw income, pulling from origin
Niklewa Jul 5, 2024
1f2783a
adding cleaning for income WIP
Niklewa Jul 5, 2024
9baf2d6
cleaning income and adjusting variable cleaner
Niklewa Jul 8, 2024
860b0b1
lint
Niklewa Jul 8, 2024
445d6d7
pulling from origin
Niklewa Jul 9, 2024
29ab1c0
cleaning ethnicity, taking clean_variable from sibling branch, lint
Niklewa Jul 9, 2024
7cbce5a
uncommenting cleaning pipeline
Niklewa Jul 9, 2024
b810bdf
fixing missing imports in the pipeline, lint
Niklewa Jul 9, 2024
b706dd3
pulling from staging-census-tracts
Niklewa Jul 9, 2024
0535f7e
pulling clean_variable from sibling branch
Niklewa Jul 9, 2024
05654c4
cleaning urbanicity, lint
Niklewa Jul 9, 2024
74c5cb0
Merge pull request #127 from BasisResearch/add-urbanicity-census-tracts
rfl-urbaniak Jul 9, 2024
d78e03b
resolving conflicts, pulling from origin
Niklewa Jul 11, 2024
8815164
resolving conflicts
Niklewa Jul 11, 2024
7e2cfd3
improving a test
Niklewa Jul 11, 2024
f5d7142
lint, fixing conflict v2
Niklewa Jul 11, 2024
dd88a61
creating the cleaning pipeline for unemployment
Niklewa Jul 12, 2024
d3230a8
cleaing industry composition
Niklewa Jul 15, 2024
d2cc37c
adding raw industry composition
Niklewa Jul 16, 2024
b2ad6c1
adding cleaning and scraping scripts
Niklewa Jul 16, 2024
a1e5e64
cleaning unemployment, lint
Niklewa Jul 18, 2024
7091438
Merge branch 'add-unemployment-rate-census-tracts' of https://github.…
Niklewa Jul 18, 2024
fc06aa2
cleaning industry, lint
Niklewa Jul 18, 2024
61d320c
add r project files to gitignore
rfl-urbaniak Jul 18, 2024
53e2c4f
rproject to gitignore
rfl-urbaniak Jul 18, 2024
6ccc985
rproject to gitignore
rfl-urbaniak Jul 18, 2024
6c63b2f
updating data guide
Niklewa Jul 23, 2024
e1bbb76
Merge branch 'add-unemployment-rate-census-tracts' of https://github.…
Niklewa Jul 23, 2024
e4582a4
updating data guide
Niklewa Jul 23, 2024
671e2d3
fromating toc of the data guide
Niklewa Jul 25, 2024
4b72e71
Merge branch 'add-unemployment-rate-census-tracts' of https://github.…
Niklewa Jul 25, 2024
1c8d9b8
modifying toc of the data guide
Niklewa Jul 25, 2024
4099f6a
pulling from origin and modifying data guide
Niklewa Jul 25, 2024
5ead286
pulling from origin
Niklewa Jul 25, 2024
d2db0cb
modifying the toc of the guide
Niklewa Jul 25, 2024
75452f3
fips query for CT WIP
Niklewa Jul 25, 2024
d937d2a
adding CTFipsQuery class and simmilarity demo for that class
Niklewa Jul 26, 2024
2eb8c54
minor corrections of the demo
Niklewa Jul 26, 2024
a4bbe12
adding tests, lint
Niklewa Jul 26, 2024
07a4ebe
adding performance test of the CTFipsQuery class
Niklewa Jul 29, 2024
f26b3f0
Merge pull request #128 from BasisResearch/add-ethnicity-census-tracts
rfl-urbaniak Aug 2, 2024
85bf4c1
Merge branch 'staging-census-tracts' of https://github.com/BasisResea…
Niklewa Aug 5, 2024
f15b02e
adding aditionall performance tests
Niklewa Aug 5, 2024
1d0e089
fixing data size mismatch WIP
Niklewa Aug 5, 2024
d962886
fixing the data mismatch issue
Niklewa Aug 8, 2024
de1347f
Merge pull request #138 from BasisResearch/add-fipsQuery-census-tracts
rfl-urbaniak Aug 14, 2024
4b1d830
adding test for missing years
Niklewa Aug 21, 2024
d2e57c1
update gitignore
rfl-urbaniak Sep 12, 2024
961f9a0
Merge branch 'main' into staging-census-tracts
rfl-urbaniak Nov 15, 2024
9798bd3
gitignore
rfl-urbaniak Nov 15, 2024
fcc605f
Merge branch 'staging-census-tracts' of https://github.com/BasisResea…
rfl-urbaniak Nov 15, 2024
0bd6ab0
types-request to setup
rfl-urbaniak Nov 15, 2024
647d83a
format lint
rfl-urbaniak Nov 15, 2024

Sorry, this diff is taking too long to generate.

It may be too large to display on GitHub.