DbtDag fails even though Airflow connection and dbt works fine

[RsFaruque]

I have a simple dbt model that creates a view in a PostgreSQL db that's hosted on docker. The dag renders but it fails without any reason. No error logs, nothing. I checked if the dbt project I am trying to schedule works on it's own and it does. The connection in Airflow is also fine, but regardless the task fails.

Below are the results of reach test. I don't really know how to debug this.

The Dag

DBT_PROFILE = ProfileConfig(
    profile_name = "dbt_profile",
    target_name = "dev",
    profile_mapping = PostgresUserPasswordProfileMapping(
        conn_id = "demo_dbt_postgres",
        profile_args = {
		    "dbname": "data_db",
		    "schema" : "public"
        }
    )   
)

dbt_dag = DbtDag(
    dag_id = "dbt_dag",
    project_config = ProjectConfig(DBT_PROJECT_PATH),
    profile_config = DBT_PROFILE,
    execution_config = DBT_EXE_CONFIG,
    start_date = datetime(2024, 11, 3),
    schedule = "@daily",
    catchup=False
)

Airflow connection test results

astro@83d49cd73fbd:/usr/local/airflow$ airflow connections test demo_dbt_postgres
[2024-11-03T17:22:05.359+0000] {configuration.py:1058} WARNING - section/key [openlineage/namespace] not found in config
[2024-11-03T17:22:06.801+0000] {plugin.py:32} WARNING - Astro managed secrets backend is disabled
Retrieving connection: 'demo_dbt_postgres'

Testing...

Connection success!

dbt debug test

...

17:27:17  Connection:
17:27:17    host: host.docker.internal
17:27:17    port: 5433
17:27:17    user: postgres
17:27:17    database: data_db
17:27:17    schema: public
17:27:17    connect_timeout: 10
17:27:17    role: None
17:27:17    search_path: None
17:27:17    keepalives_idle: 0
17:27:17    sslmode: None
17:27:17    sslcert: None
17:27:17    sslkey: None
17:27:17    sslrootcert: None
17:27:17    application_name: dbt
17:27:17    retries: 1
17:27:17  Registered adapter: postgres=1.8.2
17:27:18    Connection test: [OK connection ok]

17:27:18  All checks passed!

Dag failed

[dosubot]

Hey, @RsFaruque! I'm here to help you with any bugs, questions, or to guide you on how to contribute.

When a DbtDag fails without any error logs, it can be challenging to diagnose the issue. Here are a few steps and considerations that might help you debug the problem:

1. Check Custom Operators: The DbtDag might be using custom operators that could affect its execution. Ensure that these operators are correctly configured and that any required environment variables or configurations are set. The Cosmos repository includes several custom operators like DbtLocalBaseOperator and others that handle dbt operations ^[1][2].

2. Environment Variables and Configurations: Verify that all necessary environment variables and configurations required by the DbtDag are set in the Airflow environment. Missing configurations can lead to silent failures.

3. Dependencies and Conflicts: Ensure that all dependencies required by the dbt project are installed in the Airflow environment. There might be conflicts between certain versions of Airflow and dbt-core, especially with dependencies like jinja2 and pydantic ^[3][4][5].

4. Logging Configuration: Check if there are any specific error handling or logging configurations in the DbtDag or related operators that might suppress error messages. Adjusting the logging level or configuration might reveal more information about the failure.

5. DBT_EXE_CONFIG: Ensure that the DBT_EXE_CONFIG variable used in the DbtDag initialization is correctly set. Although the specific content of this variable wasn't found, it might be crucial for the execution of the DAG.

6. Airflow Logs: Even if the task logs are empty, check the Airflow scheduler and worker logs for any errors or warnings that might provide clues.

By addressing these areas, you might be able to identify the root cause of the issue. If the problem persists, consider simplifying the DAG to isolate the issue or testing with a minimal setup to ensure all components are functioning as expected.

To continue talking to Dosu, mention @dosu.