Documentation

https://autobricks.readthedocs.io/en/latest/

Development Setup

Create virual environment and install dependencies for local development:

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

Configuration & Authentication

Azure databricks allows the following authentication methods:

• Databricks User PAT Token
• Azure Service Principal
• Azure Service Principal over the Management End Point

This library supports all by simply setting the following environment variables as required for each method. The AUTH_TYPE sets the authorisation mode and therefore what configuration to expect. Ensure that sensitive values are managed using secret redaction e.g. key vault or some other method.

Variable	User PAT	SP	SP on Mgmt Endpoint
AUTH_TYPE	USER	SERVICE_PRINCIPAL	SERVICE_PRINCIPAL_MGMT_ENDPOINT
DBUTILSTOKEN	✓	✓	✓
TENANT_ID		✓	✓
SP_CLIENT_ID		✓	✓
SP_CLIENT_SECRET		✓	✓
AD_RESOURCE		✓	✓
MGMT_RESOURCE_ENDPOINT			✓
WORKSPACE_NAME			✓
RESOURCE_GROUP			✓
SUBSCRIPTION_ID			✓
DATABRICKS_API_HOST	✓	✓	✓

Also note that library will use simple rest calls to retrieve tokens. An alternative approach is to leverage the adal library to authenticate. If you wish to authenticate over adal please refer to following settings:

Variable	SP	SP on Mgmt Endpoint
AUTH_TYPE	SERVICE_PRINCIPAL_ADAL	SERVICE_PRINCIPAL_MGMT_ENDPOINT_ADAL
DBUTILSTOKEN	✓	✓
TENANT_ID	✓	✓
SP_CLIENT_ID	✓	✓
SP_CLIENT_SECRET	✓	✓
AD_RESOURCE	✓	✓
MGMT_RESOURCE_ENDPOINT		✓
WORKSPACE_NAME		✓
RESOURCE_GROUP		✓
SUBSCRIPTION_ID		✓
DATABRICKS_API_HOST	✓	✓

When deciding between adal and rest calls the following is relevant:

• adal may not work in Azure DevOps in highly restricted private network setups
• Rest calls are more simple however if MS makes any changes to the API's the adal isn't there to abstract those changes.

The following variables will default.

Variable	Default
AUTH_TYPE	SERVICE_PRINCIPAL
AD_RESOURCE	2ff814a6-3304-4ab8-85cb-cd0e6f879c1d
MGMT_RESOURCE_ENDPOINT	https://management.core.windows.net/

For testing, development and deployment and deployment substitute your own values between the angled brackets:

export AUTH_TYPE=USER
# export AUTH_TYPE=SERVICE_PRINCIPAL
# export AUTH_TYPE=SERVICE_PRINCIPAL_MGMT_ENDPOINT
# export AUTH_TYPE=SERVICE_PRINCIPAL_ADAL
# export AUTH_TYPE=SERVICE_PRINCIPAL_MGMT_ENDPOINT_ADAL

# required for AUTH_TYPE=USER
export DBUTILSTOKEN="<my_token>"

# required for AUTH_TYPE=SERVICE_PRINCIPAL
export TENANT_ID="<my_tenant_id>"
export SP_CLIENT_ID=<my_service_principal_client_id>
export SP_CLIENT_SECRET=<my_service_principal_secret>
export AD_RESOURCE=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d

# required for AUTH_TYPE in (SERVICE_PRINCIPAL or SERVICE_PRINCIPAL_MGMT_ENDPOINT)
export MGMT_RESOURCE_ENDPOINT=https://management.core.windows.net/
export WORKSPACE_NAME=my_dbx_workspacename
export RESOURCE_GROUP=my_dbx_resourcegroup
export SUBSCRIPTION_ID=<my_subscription_id>
export DATABRICKS_API_HOST=<my_databricks_host_url>

Exporting variables doesn't make for a great development experience so the project uses pytest.ini for mock environment variables. For example:

[pytest]
env =
    DATABRICKS_API_HOST=https://<my_databricks_host>.azuredatabricks.net
    DBUTILSTOKEN=<my_token>
    ...

REMINDER: do NOT commit pytest.ini that contains real security tokens

For convenience here is a yaml template for docker and build pipelines:

env:
    # AUTH_TYPE: USER
    AUTH_TYPE: SERVICE_PRINCIPAL
    # AUTH_TYPE: SERVICE_PRINCIPAL_MGMT_ENDPOINT
    # AUTH_TYPE: SERVICE_PRINCIPAL_ADAL
    # AUTH_TYPE: SERVICE_PRINCIPAL_MGMT_ENDPOINT_ADAL

    # required for AUTH_TYPE=USER
    # DBUTILSTOKEN: $(token)

    # required for AUTH_TYPE=SERVICE_PRINCIPAL
    TENANT_ID: $(tenant_id)
    SP_CLIENT_ID: $(service_principal_client_id)
    SP_CLIENT_SECRET: $(service_principal_secret)

    # required for AUTH_TYPE in (SERVICE_PRINCIPAL or SERVICE_PRINCIPAL_MGMT_ENDPOINT)
    WORKSPACE_NAME: $(dbx_workspace_name)
    RESOURCE_GROUP: $(dbx_resource_group)
    SUBSCRIPTION_ID: $(subscription_id)
    DATABRICKS_API_HOST: $(databricks_host_url)

Build

Build python wheel:

python setup.py sdist bdist_wheel

There is a CI build configured for this repo that builds on main origin and publishes to PyPi.

Test

Dependencies for testing:

pip install --editable .

Run tests:

pytest

Test Coverage:

pytest --cov=autobricks --cov-report=html

View the report in a browser:

./htmlcov/index.html