Enable sec10k distribution #4026
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zschira
commented
Jan 24, 2025
| @@ -42,6 +42,7 @@ ENV CONTAINER_PUDL_WORKSPACE=${CONTAINER_HOME}/pudl_work | |||
| ENV PUDL_INPUT=${CONTAINER_PUDL_WORKSPACE}/input | |||
| ENV PUDL_OUTPUT=${CONTAINER_PUDL_WORKSPACE}/output | |||
| ENV DAGSTER_HOME=${CONTAINER_PUDL_WORKSPACE}/dagster_home | |||
| ENV USE_PUDL_MODELS=True | |||
There was a problem hiding this comment.
Set environment variable to load PUDL models data during nightly builds.
zschira
commented
Jan 24, 2025
| ) | ||
| + get_pudl_models_assets() |
There was a problem hiding this comment.
Append assets to load pudl models tables to default set of assets. If USE_PUDL_MODELS is not set, then no assets will be added.
zschira
commented
Jan 24, 2025
| @@ -322,13 +322,21 @@ def load_input(self, context: InputContext) -> pd.DataFrame: | |||
| class PudlParquetIOManager(IOManager): | |||
| """IOManager that writes pudl tables to pyarrow parquet files.""" | |||
|
|
|||
| def _get_table_resource(self, table_name: str) -> Resource: | |||
There was a problem hiding this comment.
This handles metadata for PUDL models tables slightly differently from all other tables.
zschira
commented
Jan 24, 2025
| @@ -572,6 +573,24 @@ class Field(PudlMeta): | |||
| harvest: FieldHarvest = FieldHarvest() | |||
| encoder: Encoder | None = None | |||
|
|
|||
| @classmethod | |||
There was a problem hiding this comment.
PUDL models schemas are stored as pyarrow schemas, which are converted to our standard metadata structures here.
katie-lamb
approved these changes
Jan 25, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR enables the distribution of SEC10k and other future datasets produced by what will be the
pudl-modellingrepo. This repo is currently called mozilla_sec_eia and has been used to develop models for extracting company ownership data from SEC10k filings and linking this data to EIA data. This work lives separately from the main PUDL repo because training and running the models is highly resource intensive and would not interact well with the our current nightly build setup. It also allows us to isolate the complex dependencies from PUDL.Interacting with this data is fairly straightforward on the PUDL side as all that is required is to load a table and schema information from cloud storage, then write this data alongside the rest of the PUDL data. Given that outside contributors won't have credentials to access cloud resources, I've added an environment variable
USE_PUDL_MODELSthat will be checked before ever trying to load the any PUDL models data. This environment variable needs to be set for testing and nightly builds, but for most developers can be ignored.