Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add AthenaConfig to impl.py #412

Merged
merged 11 commits into from
Sep 14, 2023
Merged

chore: add AthenaConfig to impl.py #412

merged 11 commits into from
Sep 14, 2023

Conversation

SoumayaMauthoorMOJ
Copy link
Contributor

@SoumayaMauthoorMOJ SoumayaMauthoorMOJ commented Sep 11, 2023

Description

Add AthenaConfig to impl.py following recommendations

I have added the Table configuration as well as the connection parameters that can be passed to cursor.execute()

This might help to resolve #377, but it doesn't fix it by itself

Models used to test - Optional

I ran the unit tests locally and they all passed
I ran some of the functional tests and they passed too, but I didn't run all of them

Checklist

  • You followed contributing section
  • You kept your Pull Request small and focused on a single feature or bug fix.
  • You added unit testing when necessary
  • You added functional testing when necessary

@SoumayaMauthoorMOJ SoumayaMauthoorMOJ changed the title Add AthenaConfig to impl.py feat: add AthenaConfig to impl.py Sep 11, 2023
@nicor88
Copy link
Contributor

nicor88 commented Sep 11, 2023

I believe that there is still some work to do no?
In #377 we decided to pick the workgroup from model config and pass it down to adapter.run_query_with_partitions_limit_catching, this means that in each macro you need to:

  • pick the workgroup from config using jijia-> {%- set work_group = config.get('work_group', default=target.work_group) -%}
  • inject workgroup when possible when queries are executed

@SoumayaMauthoorMOJ
Copy link
Contributor Author

Hey @nicor88 you're right there is still some work to do, but I thought I would start off with a smaller change first which might help with #377. In the meantime I've raised a question on slack channel https://getdbt.slack.com/archives/CBSQTAPLG/p1694451069966319 about the best way of tackling this. I'm worried about the amount of refactoring required if I modify all the materialisations.

@nicor88
Copy link
Contributor

nicor88 commented Sep 11, 2023

@SoumayaMauthoorMOJ let's see if we get some nice suggestions, maybe you can even consider to ask in db-athena slack channel to gather mode athena experts :)

@SoumayaMauthoorMOJ
Copy link
Contributor Author

@nicor88 Did not know there was a slack channel raising now :-)

Copy link
Contributor

@svdimchenko svdimchenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the code looks nice, let's add missed parameters and docstring and we are ready to go

@SoumayaMauthoorMOJ
Copy link
Contributor Author

SoumayaMauthoorMOJ commented Sep 13, 2023

@svdimchenko I don't have permission to add the label to run the functional tests, do think it's worthwhile running those?

Also not too sure what docstring to use because explanation on dbt website isn't very clear: "database- and relation-level configs"

Hence why I raised a question on slack. I'll get the attribute descriptions from the README.

@nicor88 nicor88 added the enable-functional-tests Label to trigger functional testing label Sep 13, 2023
@svdimchenko
Copy link
Contributor

@svdimchenko I don't have permission to add the label to run the functional tests, do think it's worthwhile running those?

Also not too sure what docstring to use because explanation on dbt website isn't very clear: "database- and relation-level configs"

Hence why I raised a question on slack. I'll get the attribute descriptions from the README.

I agree with you, docstring is not obligatory. However we should have all parameters defined I guess. So this comment should be fixed: #412 (comment)

@svdimchenko svdimchenko changed the title feat: add AthenaConfig to impl.py chore: add AthenaConfig to impl.py Sep 13, 2023
Copy link
Contributor

@svdimchenko svdimchenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SoumayaMauthoorMOJ pls install and run pre-commit locally to avoid pre-commit checks fail in PR. Once it's fixed we are ready to go

@nicor88
Copy link
Contributor

nicor88 commented Sep 13, 2023

@SoumayaMauthoorMOJ it's recommended for new contributors to follow the contributing section.
In there there is a step called make setup that also takes care of installing the pre-commit hooks, therefore the same checks that we execute in the CI are executed before commiting - please have a look - exactly as @svdimchenko suggested btw.

@SoumayaMauthoorMOJ
Copy link
Contributor Author

Sorry will do this evening :-)

@svdimchenko svdimchenko merged commit bc576c4 into dbt-labs:main Sep 14, 2023
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enable-functional-tests Label to trigger functional testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature: set work_group in dbt_project.yml
3 participants