Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ISSUE] Issue with databricks_cluster and databricks_library resource - the libraries are not installed #4096

Open
Pirognoe opened this issue Oct 11, 2024 · 2 comments

Comments

@Pirognoe
Copy link

Configuration

# Copy-paste your Terraform configuration here

We have a list of libraries defined like that

databricks_libraries_cluster_247 = [ "confluent-kafka==2.5.3"
, "opencensus-ext-azure==1.1.13"
, "pymsteams==0.2.2"
, "zeep==4.1.0"
, "pandas==1.3.4"
, "lxml==4.9.1"
, "pyarrow==17.0.0"
, "openpyxl==3.1.5"
, "xmltodict==0.13.0"
, "pandarallel==1.6.5"
, "mgzip==0.2.1"
, "PureCloudPlatformClientV2==212.0.0"
, "azure-storage-queue==12.11.0"
, "parameterized==0.9.0"
, "pyyaml==6.0.2"
, "Jinja2==3.1.4"
]

and later we define the library resource like that

resource "databricks_library" "stream_hub_dh2_library" {
 for_each = toset(var.databricks_libraries_cluster_247)

 cluster_id = databricks_cluster.stream_hub_dh2.id
 pypi {
   package = each.key
 }
}

and the cluster to use those libraries

resource "databricks_cluster" "stream_hub_dh2" {
  cluster_name            = "stream_hub_dh2"
  spark_version           = data.databricks_spark_version.latest_lts.id
  node_type_id            = data.databricks_node_type.stream_worker_16Gb.id
  driver_node_type_id     = data.databricks_node_type.stream_driver_28Gb.id
  autotermination_minutes = 10
  no_wait                 = true
  autoscale {
    min_workers = 1
    max_workers = 4
  }

  spark_conf = {
    "spark.streaming.stopGracefullyOnShutdown"         : "true"
    "spark.sql.session.timeZone"                       : "UTC"
    "spark.sql.parquet.mergeSchema"                    : "true"
    "spark.hadoop.parquet.enable.summary-metadata"     : "false"
    "spark.sql.streaming.fileSink.log.compactInterval" : "999999"
    "spark.sql.sources.partitionOverwriteMode"         : "dynamic"
  }
  azure_attributes {
    availability       = "SPOT_WITH_FALLBACK_AZURE"
    first_on_demand    = 1
    spot_bid_max_price = -1
  }
}

Expected Behavior

The cluster should be created ( this happens ) and ALL libraries are assigned to it ( even if not installed immediately)

Actual Behavior

│ Error: cannot create library: Cluster is in unexpected state Pending.

│ with databricks_library.stream_hub_dh2_library["pyyaml==6.0.2"],
│ on main.tf line 140, in resource "databricks_library" "stream_hub_dh2_library":
│ 140: resource "databricks_library" "stream_hub_dh2_library" {

  1. no_wait did not seem to work properly
  2. To me it looks like if autotermination_minutes is not long enough for all libs to install ->

P.S.
Another issue is when you bump up the lib version and run tf apply - you usually end up with installation error ( cluster is displayed with 2 versions of the same lib, newer being red)

Steps to Reproduce

Terraform and provider versions

terraform {
  required_version = "~>1.7"
  required_providers {
     databricks = {
      source  = "databricks/databricks"
      version = "1.53.0"
    }
}

Is it a regression?

Debug Output

Important Factoids

Would you like to implement a fix?

@alexott
Copy link
Contributor

alexott commented Oct 11, 2024

Please remove no_wait - it was created for a specific case. Libraries could be installed only on running or terminated cluster

@Pirognoe
Copy link
Author

P.S.
Can you elaborate on what is the specific case for that? And maybe include it in docs? Anyway with or without no_wait the results seems to be the same - failure to install all the libraries and failure for TF apply

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants