Data V2

[emackev]

This PR is for developing the next version of our data pipeline. See docs/data_sources.ipynb for variables that are currently in the dataset. Desired features for new pipeline:

• Data cleaning process documented in code. Each variable has

1. high-level description in data_sources.ipynb
2. Initial cleaning pipeline: convert info from websites into wide format with GeoFIPS, GeoName, etc columns, and perform any dataset-specific cleaning. gdp_wide.csv is the canonical example. (pre-sql) The results of the initial cleaning should be stored in data/raw.
3. Final steps to make each variable compatible with the rest of our data (handle exclusions, standardize data, etc.)
   Step 2 can be the same code for all datasets (i.e., the clean_variable code). The results of the final steps should be stored in data/processed (as 4 csvs, wide vs long, and raw vs std)

• Easy to add new variables (features)
• Easy to perform relevant queries
• More variables included, based on what will be most relevant to users https://www.notion.so/basisresearch/Backend-Data-22d314e7524e4a2fb1281401c91f4545

Steps to do:

• Replace exclusions.pkl with a file included_counties.csv, with columns GeoFIPS, GeoName, Included, Explanation. Initially, leave Explanation blank -- fill it in while constructing datasets
• Make sure variable units are in data_sources.ipynb
• Some paths are hard-coded -- use find_repo_root instead.
• In included_counties.csv, for counties where Included is False, Explanation should say something like "Missing 2012 unemployment data from bls.gov"
• inspect_variable function for sanity-check plotting variables. See inspect_data.ipynb in docs/experimental_notebooks
• list_all_counties function
• Re-factor unemployment data pipeline in the way described above (currently, step 1 is in docs/experimental_notebooks/clean_unemployment.ipynb)
• Re-factor spending variable pipelines in the way described above (see docs/experimental_notebooks/grants_from_fips.ipynb for relevant notebook calls)
• Re-factor other variable pipelines in the way described above
• functions for getting additional info about a particular county e.g. grant_df_from_fips
• Dataset stored as a SQL database (connect with Ria about how to structure it)
• DataGrabber grabs from SQL database in SQL-standard ways
• Consistency tests
• Add additional variables

[emackev]

See desired variables: https://www.notion.so/basisresearch/Backend-Data-22d314e7524e4a2fb1281401c91f4545

[rfl-urbaniak]

Superseded by the current data pipeline.