Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KPO async - Invalid kube-config file. Expected key contexts in kube-config #44325

Open
1 of 2 tasks
raphaelauv opened this issue Nov 24, 2024 · 2 comments · May be fixed by #45571
Open
1 of 2 tasks

KPO async - Invalid kube-config file. Expected key contexts in kube-config #44325

raphaelauv opened this issue Nov 24, 2024 · 2 comments · May be fixed by #45571
Labels
area:core area:providers kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet provider:cncf-kubernetes Kubernetes provider related issues

Comments

@raphaelauv
Copy link
Contributor

raphaelauv commented Nov 24, 2024 •
edited
Loading

Apache Airflow version

2.10.3

What happened?

[2024-11-24, 13:59:18 UTC] {base.py:84} INFO - Retrieving connection 'kubernetes_default'
[2024-11-24, 13:59:18 UTC] {pod_manager.py:622} INFO - Pod task-two-qli8k1x6 has phase Pending
[2024-11-24, 13:59:20 UTC] {pod.py:988} INFO - Deleting pod: task-two-qli8k1x6
[2024-11-24, 13:59:20 UTC] {taskinstance.py:3311} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 767, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 733, in _execute_callable
    return ExecutionCallableRunner(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 1804, in resume_execution
    return execute_callable(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 781, in trigger_reentry
    raise AirflowException(message)
airflow.exceptions.AirflowException: Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py", line 162, in run
    state = await self._wait_for_pod_start()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py", line 223, in _wait_for_pod_start
    pod = await self.hook.get_pod(self.pod_name, self.pod_namespace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 754, in get_pod
    async with self.get_conn() as connection:
               ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 741, in get_conn
    kube_client = await self._load_config() or async_client.ApiClient()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 711, in _load_config
    await async_config.load_kube_config(
  File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/config/kube_config.py", line 603, in load_kube_config
    loader = _get_kube_config_loader_for_yaml_file(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/config/kube_config.py", line 567, in _get_kube_config_loader_for_yaml_file
    return KubeConfigLoader(
           ^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/config/kube_config.py", line 150, in __init__
    self.set_active_context(active_context)
  File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/config/kube_config.py", line 162, in set_active_context
    self._current_context = self._config['contexts'].get_with_name(
                            ~~~~~~~~~~~~^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/config/kube_config.py", line 448, in __getitem__
    raise ConfigException(
kubernetes_asyncio.config.config_exception.ConfigException: Invalid kube-config file. Expected key contexts in kube-config

How to reproduce

kind 0.24.0

kind create cluster
kind get kubeconfig --internal > conf/kube_conf

->

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: L...
    server: https://kind-control-plane:6443
  name: kind-kind
contexts:
- context:
    cluster: kind-kind
    user: kind-kind
  name: kind-kind
current-context: kind-kind
kind: Config
preferences: {}
users:
- name: kind-kind
  user:
    client-certificate-data: L...
    client-key-data: L....

airflow connection

  "kubernetes_default": {
    "conn_type": "kubernetes",
    "extra": "{\"extra__kubernetes__in_cluster\": false, \"extra__kubernetes__kube_config_path\": \"/opt/airflow/include/.kube/config\", \"extra__kubernetes__namespace\": \"default\", \"extra__kubernetes__cluster_context\": \"kind-kind\", \"extra__kubernetes__disable_verify_ssl\": false, \"extra__kubernetes__disable_tcp_keepalive\": false, \"xcom_sidecar_container_image\": \"alpine:3.16.2\"}"
  }
from airflow import DAG
from airflow.utils.dates import days_ago
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

dag = DAG(
    dag_id="kubernetes_dag",
    schedule_interval=None,
    start_date=days_ago(1),
)

with dag:
    cmd = "echo toto && sleep 4 && echo finish"

    KubernetesPodOperator(
        task_id="task-one",
        namespace="default",
        kubernetes_conn_id="kubernetes_default",
        image="alpine:3.16.2",
        cmds=["sh", "-c", cmd],
        deferrable=True,
    )

Operating System

ubuntu 22.04

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==10.0.0

Deployment

Docker-Compose

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@raphaelauv raphaelauv added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Nov 24, 2024
@dosubot dosubot bot added area:providers provider:cncf-kubernetes Kubernetes provider related issues labels Nov 24, 2024
@yovio-rca
Copy link

I get the same issue with KubernetesPodOperator with deferrable=True and in_cluster = True (default)

My Airflow runs on EKS cluster. I saw similar issue with EKSPodOperator as well in this issue: #39685

@yovio-rca
Copy link

I managed to resolve the issue by making sure to set in_cluster attribute in KubernetesPodOperator to true.

If unset the default is None and I think the subsequent logic will treat it as False.

IMHO, the behavior need to change by looking at the in_cluster value from Configuration is attribute is None.

The reason because by default in official helm chart, if we do not set in_cluster in config, it will be set to true.

When you wrote a dag, most people probably will just leave it default or unset.

Btw, the dag was working fine in "sync" mode, meaning I dont have to set in_cluster attribute and it will work fine.

@raphaelauv raphaelauv linked a pull request Jan 10, 2025 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:core area:providers kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet provider:cncf-kubernetes Kubernetes provider related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: kpo async kube_config_path raphaelauv/airflow
2 participants
@raphaelauv @yovio-rca