Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] Fix ExecutionMode.AIRFLOW_ASYNC query #1260

Open
2 tasks
tatiana opened this issue Oct 21, 2024 · 1 comment
Open
2 tasks

[bug] Fix ExecutionMode.AIRFLOW_ASYNC query #1260

tatiana opened this issue Oct 21, 2024 · 1 comment
Assignees
Labels
area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc bug Something isn't working dbt:run Primarily related to dbt run command or functionality profile:bigquery Related to BigQuery ProfileConfig stretch goal

Comments

@tatiana
Copy link
Collaborator

tatiana commented Oct 21, 2024

Context

In Cosmos 1.7, we introduced experimental BQ support to run dbt models with ExecutionMode.AIRFLOW_ASYNC in #1224 and #1230.

While chatting with @joppevos , he identified that the dbt run command:

dbt run --full-refresh

the BQ adaptor seems to create or replace on top of the table, not a drop/create:
https://github.com/dbt-labs/dbt-bigquery/blob/455c76887c9886c517df9619335066bedb1e1a43/dbt/include/bigquery/macros/adapters.sql#L16

Only if the partitions or clusters have changed then it drops
https://github.com/dbt-labs/dbt-bigquery/blob/455c76887c9886c517df9619335066bedb1e1a43/dbt/include/bigquery/macros/materializations/table.sql#L27

Action

  • Confirm this behaviour, by adding a breakpoint in the dbt run command
  • Fix the query behaviour in Cosmos
@tatiana tatiana added the bug Something isn't working label Oct 21, 2024
@dosubot dosubot bot added area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc dbt:run Primarily related to dbt run command or functionality profile:bigquery Related to BigQuery ProfileConfig labels Oct 21, 2024
@joppevos
Copy link
Contributor

Thanks for creating the issue @tatiana! I see it's assigned, but I'd love to give it a shot or collaborate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc bug Something isn't working dbt:run Primarily related to dbt run command or functionality profile:bigquery Related to BigQuery ProfileConfig stretch goal
Projects
None yet
Development

No branches or pull requests

5 participants