Build DagFactory DAGs when there is an invalid YAML in the DAGs folder

[quydx]

Before, when the Airflow DAGs folder had an invalid YAML file, no DagFactory DAGs would be loaded. This PR changes this behaviour to log any invalid YAML file paths but render valid DagFactory YAML-based DAGs.

Co-authored-by: Tatiana Al-Chueyr [email protected]

[tatiana]

@quy196hp Thank you very much for your contribution, and I am sorry for the delay in giving feedback.

Please, could you address the linting issue, so the checks pass?

==> Formatting with Black
would reformat /home/runner/work/dag-factory/dag-factory/dagfactory/dagfactory.py

Oh no! 💥 💔 💥
1 file would be reformatted, 5 files would be left unchanged.
make: *** [Makefile:38: fmt-check] Error 1

[kaxil]

The test needs fixing too:

  =========================== short test summary info ============================
  FAILED tests/test_dagfactory.py::test_load_yaml_dags_fail - Failed: DID NOT RAISE <class 'Exception'>
  ========================= 1 failed, 79 passed in 2.26s =========================

  ___________________________ test_load_yaml_dags_fail ___________________________
  
      def test_load_yaml_dags_fail():
          with pytest.raises(Exception):
  >           load_yaml_dags(
                  globals_dict=globals(),
                  dags_folder="tests/fixtures",
                  suffix=["invalid_yaml.yml"],
              )
  E           Failed: DID NOT RAISE <class 'Exception'>

[quydx]

Hi, sorrry for late response. About the failed test case, I see the test checks if there is an exception raises when loading invalid yaml files, but this PR seems to prevent the exception so can we remove this test case test_load_yaml_dags_fail @tatiana @kaxil ?

[tatiana]

@quydx I added tests in 90487b0 and e9a3c72. They are based on the PR #236

The idea is not to delete the test but to check if we're logging the error message as expected so that this type of error isn't hidden.

Additionally, I thought it was helpful to add an invalid yml to the /examples folder and guarantee that if we run airflow dags list-import-errors using that folder as the dags folder, the valid YMLs and DAGs would continue being imported without errors.

With these, the feature is fully covered with tests and we can guarantee the new behaviour is as intended.

[quydx]

Greate idea @tatiana , it seems more comprehensive way to handle this case. Things are good now