diff --git a/doc/source/cluster/kubernetes/images/grafana_ray_dashboard.png b/doc/source/cluster/kubernetes/images/grafana_ray_dashboard.png index f744e9ce8da2..3017435ae014 100644 Binary files a/doc/source/cluster/kubernetes/images/grafana_ray_dashboard.png and b/doc/source/cluster/kubernetes/images/grafana_ray_dashboard.png differ diff --git a/doc/source/cluster/kubernetes/images/grafana_ray_dashboard2.png b/doc/source/cluster/kubernetes/images/grafana_ray_dashboard2.png new file mode 100644 index 000000000000..fa5ee1e4d7ed Binary files /dev/null and b/doc/source/cluster/kubernetes/images/grafana_ray_dashboard2.png differ diff --git a/doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md b/doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md index 68c07a38870f..4265b65c5acc 100644 --- a/doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md +++ b/doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md @@ -366,8 +366,16 @@ Refer to [this Grafana document](https://grafana.com/tutorials/run-grafana-behin * TODO: Note that importing the dashboard manually is not ideal. We should find a way to import the dashboard automatically. +Once the Ray Dashboard is imported into Grafana, you can filter metrics by using the `Cluster` variable. This variable is automatically applied by default when using the provided `PodMonitor` configuration. No additional setup is needed for this labeling. + +If you have multiple RayClusters, the `Cluster` variable allows you to filter metrics specific to a particular cluster. This ensures that you can easily monitor or debug individual RayCluster instances without being overwhelmed by data from all clusters. + ![Grafana Ray Dashboard](../images/grafana_ray_dashboard.png) +Select a specific RayCluster from the dropdown to view metrics for that instance. + +![Grafana Ray Dashboard2](../images/grafana_ray_dashboard2.png) + ## Step 11: Embed Grafana panels in Ray Dashboard ```sh