Super fast snippet linting & type checking #2019
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for dlt-hub-docs canceled.
|
sh-rp
force-pushed
the
docs/improved_snippet_linting
branch
from
566b1e7
to
15f232f
Compare
burnash
reviewed
Nov 4, 2024
| @@ -90,7 +90,7 @@ The destination decorator supports settings and secrets variables. If you, for e | |||
|
|
|||
| ```py | |||
| @dlt.destination(batch_size=10, loader_file_format="jsonl", name="my_destination") | |||
| def my_destination(items: TDataItems, table: TTableSchema, api_key: dlt.secrets.value) -> None: | |||
| def my_destination(items: TDataItems, table: TTableSchema, api_key: str = dlt.secrets.value) -> None: | |||
burnash
reviewed
Nov 4, 2024
| @@ -200,7 +202,7 @@ If you wish to create your own pipelines, you can leverage source and resource m | |||
| base_id = "Please set me up!" # The ID of the base. | |||
|
|
|||
| airtables = airtable_source(base_id=base_id) | |||
| load_info = pipeline.run(load_data, write_disposition="replace") | |||
| load_info = pipeline.run(airtables, write_disposition="replace") | |||
burnash
reviewed
Nov 4, 2024
| @@ -236,7 +236,7 @@ To create your data pipeline using single loading and [incremental data loading] | |||
| 1. To load data from a specific date, including dependent endpoints: | |||
|
|
|||
| ```py | |||
| load_data = workable_source(start_date=datetime(2022, 1, 1), load_details=True) | |||
| load_data = workable_source(start_date=DateTime(2022, 1, 1), load_details=True) | |||
There was a problem hiding this comment.
Here, the type is correct, but now it's not clear where DateTime is coming from. Maybe it'd be better to explicitly use pendulum.DateTime?
burnash
previously approved these changes
Nov 4, 2024
sh-rp
force-pushed
the
docs/improved_snippet_linting
branch
from
8cc075a
to
b84e50c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Lints and typechecks all docs snippets in one go. For this all snippets of each docs page are rendered into one python file and all those files have the same structure as the docs pages. Then the lint and typechecking commands are run on that full folder tree in one command each.
There is one benefit to this that I thought of when writing the dataset docs: For snippets on one docs page that relate to each other (which is a good way of writing pages anyway imho) we can typecheck across them and also make sure that variables stay the same type across one docs page. I think this will need a bit of more work from me though.
ToDo: