Skip to content

Latest commit

 

History

History
96 lines (71 loc) · 3.12 KB

README.rst

File metadata and controls

96 lines (71 loc) · 3.12 KB
Raw

airflow-declarative

Airflow declarative DAGs via YAML.

Compatibility:

  • Python 2.7 / 3.5+
  • Airflow 1.10.4+

Key Features

  • Declarative DAGs in plain text YAML helps a lot to understand how DAG will looks like. Made for humans, not programmers.
  • It makes extremely hard to turn your DAGs into code mess. Even if you make complicated YAMLs generator the result would be readable for humans.
  • No more guilty about coupling business logic with task management system (Airflow). They now could coexists separated.
  • Static analysis becomes a trivial task.
  • It's a good abstraction to create your own scheduler/worker compatible with original Airflow one.

Examples

Check tests/dags directory for example of DAGs which will works and which won't. Use src/airflow_declarative/schema.py module for the reference about YAML file schema. It should be self descriptive.

Don't be shy to experiment: trafaret-config will help you to understand what had gone wrong and why and where.

Usage

We provide support for two installation options:

  1. As a complementary side package for the upstream Airflow.
  2. As a built-in Airflow functionality using patches for Airflow.

Upstream Airflow

The idea is to put a Python script to the dags_folder which would load the declarative dags via airflow_declarative. More details: Installation using Upstream Airflow.

import os

import airflow_declarative

# Assuming that the yaml dags are located in the same directory
# as this Python module:
root = os.path.dirname(__file__)

dags_list = [
    airflow_declarative.from_path(os.path.join(root, item))
    for item in os.listdir(root)
    if item.endswith((".yml", ".yaml"))
]

globals().update({dag.dag_id: dag for dags in dags_list for dag in dags})

Patched Airflow

We provide ready to use patches in the patches directory. To use them you will need to apply a patch to a corresponding Airflow version and then build it yourself. More details: Installation using Patched Airflow.