Skip to content

Commit

Permalink
Create databricks-changes.md
Browse files Browse the repository at this point in the history
  • Loading branch information
benc-db authored Oct 25, 2024
1 parent 94ae62a commit a5a0114
Showing 1 changed file with 29 additions and 0 deletions.
29 changes: 29 additions & 0 deletions website/docs/reference/global-configs/databricks-changes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: "Databricks adapter behavior changes"
id: "databricks-changes"
sidebar: "Databricks"
---

## Behavior Changes

Here are the current [behavior change flags](/docs/reference/global-configs/behavior-changes.md#behavior-change-flags) that are specific to `dbt-databricks`:

| Flag | `dbt-databricks`: Intro | `dbt-databricks`: Maturity |
| ----------------------------- | ----------------------- | -------------------------- |
| `use_info_schema_for_columns` | 1.9.0 | TBD |
| `use_user_folder_for_python` | 1.9.0 | TBD |

### Use information schema for columns

The `use_info_schema_for_columns` flag is `False` by default.
Setting this flag to `True` will use `information_schema`, rather than `describe extended`, to get column metadata for Unity Catalog tables.
This setting is to avoid issues where `describe extended` truncates type information when the type is a complex struct;
however, this setting is not ready to be default behavior, as in order to ensure that `information_schema` is complete, we must run `REPAIR TABLE {{relation}} SYNC METADATA` prior to querying, which adds overhead.
This flag cannot be considered for maturity until we no longer need to add this overhead to every incremental materialization.

### Use user's folder for Python model notebooks

The `use_user_folder_for_python` flag is `False` by default, and results in writing uploaded python model notebooks to `/Shared/dbt_python_models/{{schema}}/`.
Setting this flag to `True` will instead write notebooks to `/Users/{{current user}}/{{catalog}}/{{schema}}/`
Writing to the `Shared` folder is being deprecated by Databricks as not following governance best practices.
We plan to promote this flag to maturity in 1.10.0.

0 comments on commit a5a0114

Please sign in to comment.