Prometheus input on On-Premise servers

[rifftual]

The documentation doesn't talk about running CloudWatch agent with Prometheus input plugin on On-Premise servers.

I tried to use the plugin with the agent. The logs indicate that the plugin is loaded but there is nothing about Prometheus in the logs after that:

OS: Ubuntu 20.04 LTS_
CloudWatch agent version (DPKG): 1.247347.6b250880-1

There is a HTTP service running on localhost:5000 which exports metrics in Prometheus' format.

CloudWatch agent configuration:

{
  "agent": {
    "metrics_collection_interval": 10,
    "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
    "region": "eu-central-1",
    "debug": true
  },
  "logs": {
    "metrics_collected": {
      "prometheus":{
              "log_group_name": "prometheus",
              "prometheus_config_path": "/etc/amazon/amazon-cloudwatch-agent/prometheus.yaml",
              "emf_processor": {
                "metric_declaration_dedup": true,
                "metric_namespace": "CWAgent-Prometheus",
                "metric_unit": {
                        "jvm_threads_number": "Count"
                },
                "metric_declaration": [
                        {
                                "source_labels": ["job","key1"],
                                "label_matcher": "MY_JOB;^value1",
                                "dimensions": [
                                        ["key1"]
                                ],
                                "metric_selectors": [
                                        "^jvm_threads_number$"
                                ]
                        }
                ]
              }
      }
    },
    "log_stream_name": "test",
    "force_flush_interval" : 15
  }
}

prometheus.yaml

global:
        scrape_interval: 1m
        scrape_timeout: 10s
        scrape_configs:
                - job_name: MY_JOB
                  sample_limit: 10000
                  file_sd_configs:
                          - files: ["/etc/amazon/amazon-cloudwatch-agent/prometheus_sd_1.yaml"]

prometheus_sd_1.yaml

- targets:
        - 127.0.0.1:5000
  labels:
        - key1: value1

Log

2021/05/28 10:17:30 I! 2021/05/28 10:17:30 E! ec2metadata is not available
I! Detected the instance is OnPrem
2021/05/28 10:17:30 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it.
2021/05/28 10:17:30 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_tmp.json ...
Valid Json input schema.
I! Detecting run_as_user...
Got Home directory: /root
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded

2021/05/28 10:17:30 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2021/05/28 10:17:30 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2021/05/28 10:17:30 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_tmp.json ...
2021/05/28 10:17:30 I! Detected runAsUser: root
2021/05/28 10:17:30 I! Changing ownership of [/opt/aws/amazon-cloudwatch-agent/logs /opt/aws/amazon-cloudwatch-agent/etc /opt/aws/amazon-cloudwatch-agent/var] to root:root
2021-05-28T07:17:30Z I! Starting AmazonCloudWatchAgent 1.247347.6
2021-05-28T07:17:30Z I! Loaded inputs: prometheus_scraper
2021-05-28T07:17:30Z I! Loaded aggregators:
2021-05-28T07:17:30Z I! Loaded processors: emfProcessor
2021-05-28T07:17:30Z I! Loaded outputs: cloudwatchlogs
2021-05-28T07:17:30Z I! Tags enabled: host=foobar
2021-05-28T07:17:30Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"foobar", Flush Interval:1s
2021-05-28T07:17:30Z D! [agent] Initializing plugins
2021-05-28T07:17:30Z D! [agent] Connecting outputs
2021-05-28T07:17:30Z D! [agent] Attempting connection to [outputs.cloudwatchlogs]
2021-05-28T07:17:30Z D! [agent] Successfully connected to outputs.cloudwatchlogs
2021-05-28T07:17:30Z D! [agent] Starting service inputs
2021-05-28T07:17:30Z I! [logagent] starting
2021-05-28T07:17:30Z I! [logagent] found plugin cloudwatchlogs is a log backend
2021-05-28T07:17:31Z D! [outputs.cloudwatchlogs] Buffer fullness: 0 / 10000 metrics

The last line repeats indefinitely.

[pxaws]

Could you change the following line in the agent config:

"label_matcher": "MY_JOB;^value1",

to

"label_matcher": "MY_JOB;value1",

It's recommended not to use ^ if you have multiple label regex. See label_matcher in https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-configure.html#ContainerInsights-Prometheus-Setup-cw-agent-config2 for details.

label_matcher is a regular expression that checks the value of the labels listed in source_labels. The metrics that match are enabled for inclusion in the embedded metric format sent to CloudWatch.

If you have multiple labels specified in source_labels, we recommend that you do not use ^ or $ characters in the regular expression for label_matcher.

[rifftual]

Didn't help. Also, I find it very weird that there aren't any logs about Prometheus except for the plugin load. I checked the source code for the Prometheus input plugin and it does log messages.

[rifftual]

I don't what was the problem but the scraping works now. I'm guessing mismatch between referring to .yaml files and having .yml files in the filesystem

[pingleig]

Due to our bug #209 you will actually see all the metrics in agent's log ... It's fixed in main branch but the binary release is not out yet.