Kbauer/add all on host assertions

[kb-newrelic]

Summary

• Migrated alert-based tests defined in test/canaries/alerts/vars.auto.tfvars.dist to hostmetrics_slow_test.go
  • Changes
    • instead of comparison testing with previous version, we evaluate against a static threshold to prevent steady but undetectable regression
    • removed host receiver disk.operations write duplicate
    • removed mountpoint where clause from system.filesystem.inodes.usage and system.filesystem.usage assertions as no metric with mountpoint / was present which is most likely due to github runner having a different file system layout or collector not yet running with the correct permissions. Receiver docs didn't provide much insight.
  • It's hard to predict exact values for the metrics, so I kept it simple with >0 (default) and >=0 (for metrics that have no data unless we'll generate some artificially, e.g. system.network.dropped) until we have a more stable way to artificially generate signals that the receiver would harvest
• Increased per-test timeout as there doesn't seem to be a test-specific setting in golang without extra dependencies
• Pod hostname is set programmatically by setting OTEL_RESOURCE_ATTRIBUTES, otherwise thehost.name variable when running tests in CI has an unpredictable value based on the underlying VM (but different than the hostname returned on the test machine), so NRQL queries would fail

Test

• Manually kicked off nightly build succeeded

Notes

• Ideally we would implement a rate-limited http client for the tests to not hit the 25req/s rate limit of NerdGraph. I opted for a simple sleep-based approach to avoid that overhead for now.
• Test currently sends one NRQL request for each metric to be tested. Especially for the metrics that get queried multiple times with different attribute where clauses, this is somewhat wasteful. I decided not to optimize too much too soon though as the shape of the tests might change drastically, but it is something to keep in mind.