Skip to content

Commit

Permalink
feat: Default reattach_on_restart parameter to True in bigquery_etl_q…
Browse files Browse the repository at this point in the history
…uery function (#1961)

We've been hit (again) by Airflow losing track of a running pod, restarting a task and failing with `More than one pod running with labels ...` error.
We previously set some of the tasks to reattach to running pods on restart ([1], [2], [3]) which solved this problem for them. We didn't set it by default for all tasks back then to avoid situations where not-autodeleted pods are stuck with the old code. Now that DEs have permissions required to delete pods it seems safe to revisit this.

[1] #1820
[2] #1834
[3] #1837
  • Loading branch information
akkomar authored Apr 8, 2024
1 parent 09acddb commit 3234be5
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion utils/gcp.py
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ def bigquery_etl_query(
arguments=(),
project_id=None,
sql_file_path=None,
reattach_on_restart=False,
reattach_on_restart=True,
gcp_conn_id="google_cloud_airflow_gke",
gke_project_id=GCP_PROJECT_ID,
gke_location="us-west1",
Expand Down

0 comments on commit 3234be5

Please sign in to comment.