[prometheus] agent is printing all the scraped metrics in log

[pingleig]

Overview

The latest version of docker image is printing all the metrics into its own log, the root cause is in #193 where @pingleig forgot to remove the debug statement ae64a26#r50943644

amazon-cloudwatch-agent/plugins/inputs/prometheus_scraper/metrics_receiver.go

Line 86 in c7a6db5

log.Printf("%s %v", pm.metricName, labelMap)

What should you do

• If you are only using prometheus part of the agent, you can safely revert back to previous version unless you are relabelling job, instance or __name__ (see Versions section for yaml and docker image you should use).
• If you are not using prometheus part of the agent or not collecting log of agent itself, you won't get billed for the large amount (useless) log, though it will add some pressure on the node running agent depends the volume of prometheus metrics.
• If you are using agent for collecting container metrics on EKS, agent is using cadvisor and not using prometheus. And if you uses contianerd, only the latest release contains [k8s] Pod metrics is gone when using containerd as runtime #188 so you should stick with the lastest version.

Versions

Previous version

• Manifest: (1.3.5) https://github.com/aws-samples/amazon-cloudwatch-container-insights/releases/tag/k8s%2F1.3.5
• Version: 1.247347.5b250583
• Image: amazon/cloudwatch-agent:1.247347.5b250583 or public.ecr.aws/cloudwatch-agent/cloudwatch-agent:1.247347.5b250583

Latest version (w/ the log bug)

• Manifest: (1.3.6) https://github.com/aws-samples/amazon-cloudwatch-container-insights/releases/tag/k8s%2F1.3.6
• Version: 1.247347.6b250880
• Image: amazon/cloudwatch-agent:1.247347.6b250880 or public.ecr.aws/cloudwatch-agent/cloudwatch-agent:1.247347.6b250880

[pingleig]

Fixed in latest release

• Manifest (1.3.7): https://github.com/aws-samples/amazon-cloudwatch-container-insights/releases/tag/k8s%2F1.3.7
• Version: 1.247348.0b251302
• Image: amazon/cloudwatch-agent:1.247348.0b251302 or public.ecr.aws/cloudwatch-agent/cloudwatch-agent:1.247348.0b251302