This is the home of dynatrace-gcp-function which provides the mechanism to pull Google Cloud metrics into Dynatrace.
To help with function deployment you can use automation scripts available in this repo.
Maintaining its lifecycle places a burden on the operational team.
dynatrace-gcp-function is a Cloud Function written in Python that pulls metrics for configured services from Cloud Monitoring API. Function execution is triggered by Pub/Sub upon 1 minute schedule defined in Cloud Scheduler. Authentication token to query for metrics is retrieved for the scope of Service account that is created during installation process. Once the time series are collected, the values are pushed to Dynatrace Metrics API v2 using Dynatrace API token and URL to Dynatrace tenant environment stored in Secret Manager.
In addition to metrics dynatrace-gcp-function is calling Service specific API's (for example Pub/Sub API). The purpose is to get properties of the instances that are not available in Monitoring API. Particularly the function try to retrieve endpoint addresses (FQDN's, IP addresses).
Architecture with Google Cloud Function deployment
As alternative to Cloud Function deployment it's possible to run monitoring as Kubernetes Container. In this case all configurations and secretes are stored as K8S ConfigMap / Secretes objects.
Architecture with Google Cloud Function deployment
| Google Cloud service | Metric pulling | Pre-defined dashboards | Pre-defined alerts |
|---|---|---|---|
| Google Cloud APIs | Y | Y | N |
| Google Cloud Function | Y | Y | N |
| Google Cloud SQL | Y | Y | N |
| Google Cloud Datastore | Y | Y | N |
| Google Cloud Filestore | Y | Y | N |
| Google Cloud Storage | Y | Y | N |
| Google Cloud Load Balancing | Y | Y | N |
| Google Cloud Pub/Sub | Y | Y | N |
| Google Kubernetes Node | Y | Y | N |
| Google Kubernetes Container | Y | Y | N |
| Google Kubernetes Cluster | Y | Y | N |
| VM Instance | Y | N | N |
Make sure the following dependencies are installed:
For example, You can use:
sudo wget https://github.com/mikefarah/yq/releases/download/3.4.0/yq_linux_amd64 -O /usr/bin/yq && sudo chmod +x /usr/bin/yq
dynatrace-gcp-function uses Cloud Scheduler that requires App Engine to be created. If you don't have App Engine enabled yet, installer script will prompt you to Create it and select region, where it will run. Reference: Cloud Scheduler documentation
Download & run the dynatrace-gcp-function installation script
wget https://raw.githubusercontent.com/dynatrace-oss/dynatrace-gcp-function/master/scripts/setup.sh -O setup.sh ; chmod a+x setup.sh ; ./setup.sh
Installation script will prompt for following parameters:
| Parameter | Description |
|---|---|
| GCP project | Google Cloud project, where dynatrace-gcp-function should be deployed to. By default, current project set for gcloud CLI. |
| Function size | Amount of memory that should be assigned to the function. Possible options [s] - small, up to 500 instances, 256 MB memory allocated to function [m] - medium, up to 1000 instances, 512 MB memory allocated to function [l] - large, up to 5000 instances, 2048 MB memory allocated to function Please note that You will be able to adjust amount of memory after installation. |
| Dynatrace tenant URI | The URL to Your Dynatrace SaaS or Managed environment |
| Dynatrace API token | Dynatrace API token. You can learn how to generate token Dynatrace API - Tokens and authentication manual. Integration requires API v2 Ingest metrics, API v1 Read configuration and WriteConfig Token permission. |
Make sure the following dependencies are installed:
- jq
- yq mikefarah/yq GitHub project
- Google Cloud SDK Google Cloud SDK installer
For example, on Ubuntu You can use:
sudo wget https://github.com/mikefarah/yq/releases/download/3.4.0/yq_linux_amd64 -O /usr/bin/yq && sudo chmod +x /usr/bin/yq
sudo apt-get install jq
curl https://sdk.cloud.google.com | bash
Restart the console and initialize Cloud SDK (Initializing Cloud SDK):
gcloud init
Run the dynatrace-gcp-function installation script:
wget https://raw.githubusercontent.com/dynatrace-oss/dynatrace-gcp-function/master/scripts/setup.sh -O setup.sh ; chmod a+x setup.sh ; ./setup.sh
Installation script will prompt for following parameters:
| Parameter | Description |
|---|---|
| GCP project | Google Cloud project, where dynatrace-gcp-function should be deployed to. By default, current project set for gcloud CLI. |
| Function size | Amount of memory that should be assigned to the function. Possible options [s] - small, up to 500 instances, 256 MB memory allocated to function [m] - medium, up to 1000 instances, 512 MB memory allocated to function [l] - large, up to 5000 instances, 2048 MB memory allocated to function Please note that You will be able to adjust amount of memory after installation. |
| Dynatrace tenant URI | The URL to Your Dynatrace SaaS or Managed environment |
| Dynatrace API token | Dynatrace API token. You can learn how to generate token Dynatrace API - Tokens and authentication manual. Integration requires Ingest metrics using API V2 Token permission. |
Please note dynatrace-gcp-function uses Cloud Scheduler that requires App Engine to be created. If you don't have App Engine enabled yet, installer script will prompt you to Create it and select region, where it will run. Reference: Cloud Scheduler documentation
- Google Cloud SDK Google Cloud SDK installer
- Kubernetes CLI Install and setup kubectl
Create dynatrace namespace with kubectl, and secrets for Dynatrace cluster API token and URL.
Replace {DYNATRACE_URL} with URL to Your Dynatrace SaaS or Managed environment.
Replace {DYNATRACE_API_TOKEN} with Dynatrace API token. You can learn how to generate token Dynatrace API - Tokens and authentication manual. Integration requires Ingest metrics using API V2 Token permission.
kubectl create namespace dynatrace
kubectl -n dynatrace create secret generic dynatrace-gcp-function-secret --from-literal="access-key={DYNATRACE_API_TOKEN}" --from-literal="url={DYNATRACE_URL}"
Create IAM Service Account with Cloud Shell and configure it for workload identity. Replace {GCP-PROJECT-ID} with your GCP project ID
gcloud iam service-accounts create dynatrace-gcp-function-sa
gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:{GCP-PROJECT-ID}.svc.id.goog[dynatrace/dynatrace-gcp-function-sa]" dynatrace-gcp-function-sa@{GCP-PROJECT-ID}.iam.gserviceaccount.com
Grant required IAM policies to Service Account. Replace {GCP-PROJECT-ID} with your GCP project ID
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/monitoring.editor
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/monitoring.viewer
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/compute.viewer
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/cloudsql.viewer
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/cloudfunctions.viewer
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/file.viewer
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/pubsub.viewer
Enable API's required for monitoring
gcloud services enable cloudapis.googleapis.com monitoring.googleapis.com cloudresourcemanager.googleapis.com
Download and install dynatrace-gcp-function.yaml Kubernetes objects:
wget https://raw.githubusercontent.com/dynatrace-oss/dynatrace-gcp-function/master/k8s/dynatrace-gcp-function.yaml
You can adjust the function behavior in dynatrace-gcp-function-config Config Map defined in dynatrace-gcp-function.yaml.
Deploy Kubernetes objects:
kubectl apply -f dynatrace-gcp-function.yaml
Create annotation for service account. Replace {GCP-PROJECT-ID} with your GCP project ID:
kubectl annotate serviceaccount --namespace dynatrace dynatrace-gcp-function-sa iam.gke.io/gcp-service-account=dynatrace-gcp-function-sa@{GCP-PROJECT-ID}.iam.gserviceaccount.com
Check the container status:
kubectl -n dynatrace logs -l app=dynatrace-gcp-function
Logs should output overall status of each metrics pull execution:
Starting download of metadata for service 'cloudsql_database'
Download of metadata for service 'pubsub_subscription' finished
Download of metadata for service 'cloudsql_database' finished
Download of metadata for service 'filestore_instance' finished
Download of metadata for service 'cloud_function' finished
Fetched GCP data in 1.1237902641296387 s
Ingest response: {'linesOk': 421, 'linesInvalid': 0, 'error': None}
Finished uploading metric ingest lines to Dynatrace in 0.36574411392211914 s
Pushing self monitoring time series to GCP Monitor...
Finished pushing self monitoring time series to GCP Monitor
It's possible to deploy dynatrace-gcp-function to push metrics to Dynatrace from multiple GCP projects.
For example: it's possible to run function in project dedicated for monitoring and get metrics from production/stage/dev projects. The concept is illustrated on the diagram below:
Sample multi-project deployment
To configure multi-project support in this example You will need:
- Running
dynatrace-gcp-function(Cloud Function or Kubernetes) - IAM service account configured with
dynatrace-gcp-function(for exampledynatrace-gcp-function-sa@{GCP-FUNCTION-PROJECT-ID}.iam.gserviceaccount.com)
Now You need to grant required IAM policies to Service Account for desired projects (for example PROJECT-A, PROJECT-B, PROJECT-C).
Replace {GCP-PROJECT-ID-TO-MONITOR} with the project You wish to enable monitoring for. Replace {GCP-FUNCTION-PROJECT-ID} with the project the Service Account is created on / function is deployed. Repeat the step for all projects.
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID-TO-MONITOR} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-FUNCTION-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/monitoring.editor
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID-TO-MONITOR} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-FUNCTION-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/monitoring.viewer
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID-TO-MONITOR} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-FUNCTION-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/compute.viewer
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID-TO-MONITOR} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-FUNCTION-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/cloudsql.viewer
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID-TO-MONITOR} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-FUNCTION-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/cloudfunctions.viewer
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID-TO-MONITOR} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-FUNCTION-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/file.viewer
gcloud projects add-iam-policy-binding {GCP-PROJECT-ID-TO-MONITOR} --member="serviceAccount:dynatrace-gcp-function-sa@{GCP-FUNCTION-PROJECT-ID}.iam.gserviceaccount.com" --role=roles/pubsub.viewer
After next monitoring run (~1 minute) metrics from configured projects should start appearing in Dynatrace.
Alternatively You may grant access for all projects in IAM & Admin console in Google Cloud Portal. Simply navigate to all of the projects and add permissions for Service Account attached to function:
Installation script deploys Dynatrace Dashboards for GCP services within provided Dynatrace tenant. You can browse them in Dynatrace UI.
Dashboards list:
Sample dashboard for GCP API:
dynatrace-gcp-function reports self-monitoring metrics as Google Metrics. This allow to track eventual problems with communication between the function and Dynatrace cluster.
Self monitoring metrics:
| Metrics | Description |
|---|---|
| MINT lines ingested | Amount of lines of metrics (data points) ingested into Dynatrace Metric INTerface for given interval. |
| Dynatrace connectivity | Status (1 -ok) indicating the connectivity between monitoring function and Dynatrace cluster. Connectivity can be broken due to wrong Dynatrace URL provided, wrong API token or network connectivity issues |
| Dynatrace failed requests count | Amount of requests that were rejected by Dynatrace Metric INTerface. The reason for failure can be unexpected data point value or reaching request quota for metric ingest |
| Dynatrace requests count | Amount of requests that were send to Dynatrace cluster. |
Installation script deploy dynatrace-gcp-function Self Monitoring dashboard upon installation process to help tracking health of the solution.
See CONTRIBUTING for details on submitting changes.
dynatrace-gcp-function is under Apache 2.0 license. See LICENSE for details.