v0.9.0

What's Changed

This is the first release for brickflow, which is a tool to build Databricks workflows using python. Right now brickflow supports CDKTF(to be deprecated soon) and Databricks Asset Bundles.

• Moved the source code from Internal Nike repository to the OSS public repo contributed by @asingamaneni & @stikkireddy
• Github actions initial setup by @stikkireddy in #3
• Multiple changes by @stikkireddy in #2
  • commit for multi projects and reimporting existing workflows with same name
  • fixed tests and added support for notebook native tasks
  • added some comments and examples of how to use parameters.
  • changes to help string for project add and also resolving recursively to root from notebook context as well.
  • added /Workspace root for getting notebook path
  • fixed removing projects correctly from the multi-project.yml file as well
  • fixed tests and added support for multiple project deployments in the same directory by making the environment names different values
  • be lax with development for local development if git commit yields nothing
  • added support for workflow level notifications and webhooks, task notification settings and support for photon in compute. also root, filepath and statepath will write posix paths. python binary will be fetched from entrypoint executable that is used
  • fixed unnecessary code in plugins and added a feature flag for using project name as part of the env
  • added synth command for projects
  • added local task settings at task level and also added workflow level settings for webhooks, emails and notifications
  • suffixed all the workflow level settings with Workflow due to import from brickflow making it confusing with task level objects
  • make sure import manager uses the new import folder syntax structure.
  • make sure imports only work with projects tagged with this brickflow project, permissive of duplicates still
  • clean up logging entry points to only log when project is kicked off or cli is called.
  • clear docs on resolver for imports and proper docs
  • remove resolver and properly load plugins for airflow and py4j (cronhelper) integration
  • made cronhelper lazy, added unit tests for plugins and added py4j as a dev dependency
  • added setup for java for github actions
  • automatically add the brickflow library to the cluster and flag to opt out. Run bundles cli from anywhere within the project root
  • set the monorepo path variable for correct bundle paths
  • updated the ascii art to be more legible
  • fixed bug related to suffix and prefix not being properly populated during runtime.
  • update databricks cli to 0.203.0
  • fixed library handling
  • update all deps
  • separate airflow installation from brickflow and remove it as optional
  • added deployment job to pypi and fixed issue with code coverage
• Add logos and update README.md by @jhollow in #6
• Adding github badges by @asingamaneni in #5

New Contributors

• @stikkireddy made their first contribution in #3
• @jhollow made their first contribution in #6
• @asingamaneni made their first contribution in #5

Full Changelog: https://github.com/Nike-Inc/brickflow/commits/v0.9.0