Add observability samples (self managed) for Istio addon

[biefy]

Add observability samples (self managed) for Istio addon.
The yaml files are modified based on https://github.com/istio/istio/blob/master/samples/addons/.

[chasewilson]

Minor updates for grammar and readability.

[shashankbarsin]

• One problem of having the sample deployment of prometheus and grafana itself in the samples is that we'd need to keep tabs on when newer versions of prometheus/grafana are available and then update these samples (without which deployment of older samples could result in vulnerabilities). To address this, what if we instead document a step each on self-installing Prometheus or Grafana with reference links to their docs (for example, https://github.com/prometheus-operator/kube-prometheus/tree/main?tab=readme-ov-file#quickstart) and remove the sample deployment files?
• I observed that istio grafana dashboards from community are mentioned directly as a configmap. It'd again be the same problem as above where we'd need to maintain them over time. Instead, can we point to istio community grafana dashbaords here (https://github.com/istio/istio/tree/master/manifests/addons/dashboards) and document steps on how to import these dashboards into Grafana after it's deployed?
• The one part I think we should retain is the scrape config

[biefy]

• One problem of having the sample deployment of prometheus and grafana itself in the samples is that we'd need to keep tabs on when newer versions of prometheus/grafana are available and then update these samples (without which deployment of older samples could result in vulnerabilities). To address this, what if we instead document a step each on self-installing Prometheus or Grafana with reference links to their docs (for example, https://github.com/prometheus-operator/kube-prometheus/tree/main?tab=readme-ov-file#quickstart) and remove the sample deployment files?
• I observed that istio grafana dashboards from community are mentioned directly as a configmap. It'd again be the same problem as above where we'd need to maintain them over time. Instead, can we point to istio community grafana dashbaords here (https://github.com/istio/istio/tree/master/manifests/addons/dashboards) and document steps on how to import these dashboards into Grafana after it's deployed?
• The one part I think we should retain is the scrape config

These are great questions.

For refreshing these files, by looking through the upstream files history, they mostly refresh images. We could do the same - maybe twice a year?

For the scraping configs, they are directly from the upstream files with some namespace replacement.

[microsoft-github-policy-service]

@Azure/aks-pm issue needs labels