Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not sure why response_result_data_Hardware_HDD_used_by_node is used as a label in prometheus metrics #61

Open
pruneau628 opened this issue Apr 5, 2023 · 12 comments

Comments

@pruneau628
Copy link

It artificially creates different times series, is there a rationale for this ?

@WadeBarnes
Copy link
Member

There are a number of text fields returned by the call to validator-info that we want to display. Prometheus does not handle text, so adding the text fields to labels allows the text to be accessed in Prometheus queries. This was done when we were initially evaluating Prometheus and Influx.

@pruneau628
Copy link
Author

I do understand the purpose, and I can get behind including attributes that seldom changes, like IP configuration, OS version, but the disk space usage change a bit too often to qualify as a label, though. It also changes randomly, which makes it even worse as a label.

@WadeBarnes
Copy link
Member

It would be better to handle text fields through Influx and remove these labels from Prometheus, which brings up a question. Do we use Prometheus and Influx, or just Influx? Prometheus makes collecting and graphing metrics super easy, but it does not handle text. Influx is more difficult when it comes to queries, but supports text and metrics. Or is there another solution? @ConnorBarnes88, do you have any thoughts on this?

@pruneau628
Copy link
Author

pruneau628 commented Apr 6, 2023

  • from a restricted point of view, response_result_data_Hardware_HDD_used_by_node is actually a number, and should be treated as such
  • removing prometheus from the equation creates a void with respect of raising alarms.
  • it could also make sense to use this monitoring stack to gather metrics from agents and other parts of the eosystem, and prometheus is still a good candidate.

Now, I have the feeling I'm missing the big picture here.

@WadeBarnes
Copy link
Member

WadeBarnes commented Apr 6, 2023

  • from a restricted point of view, response_result_data_Hardware_HDD_used_by_node is actually a number, and should be treated as such

The issue is we receive it as text in human readable form where the number could be in MBs, GBs, etc:

"Hardware": {
  "HDD_used_by_node": "1037 MBs"
},

Converting that into an actual number will require some transformation as the telegraf level. Converting it to a number would allow us to remove it as a label; stop treating it as text.

  • removing prometheus from the equation creates a void with respect of raising alarms.
  • it could also make sense to use this monitoring stack to gather metrics from agents and other parts of the eosystem, and prometheus is still a good candidate.

Agreed

Now, I have the feeling I'm missing the big picture here.

I don't think you are.

@pruneau628
Copy link
Author

Thanks for your trust @WadeBarnes.
I have been looking into my own affirmation about prometheus, and it actually looks like the latest influxdb oss version 2.7.0 support alarms as well.
If prometheus is still perceived as useless, this is probably the way to go.
But that's just from reading the docs, not experimentation yet.

@ConnorBarnes88
Copy link
Contributor

ConnorBarnes88 commented Apr 6, 2023 via email

@pruneau628
Copy link
Author

Wow, I was not expecting a simple question to devovle into a full-on architecture review.
I'm definitely interested in contributing, but not sure how to go about it.

@WadeBarnes
Copy link
Member

lol, you picked the right (or wrong) question @pruneau628. For this metric I think we could do a conversion so Prometheus can consume it more easily. However there are many other data elements from the validator-info calls that we're interested in and are in text form. So we still have a few choices to make. Stick with a mix of Prometheus and Influx and use each for their strengths, or switch to Influx completely. Both choices introduce a certain level of complexity, the question is which will be easier to maintain in the log run.

Perhaps you could assist with the decisions by doing a bit more investigation into the capabilities of Influx 2.x.

@pruneau628
Copy link
Author

Ok, we are doing just that, and will report on the progress as soon as possible.

@GuillaumeBourque-QC
Copy link
Contributor

GuillaumeBourque-QC commented Aug 6, 2024

Hello all,

I'm very new to all this, but if I may add to the discussion, why not use Grafana for the alerting part ? Having run a couple of influxdb server in production with telegraf and Grafana for about 5 years without a single issue, I would say that I don't see the complexity in using these 3 products ( I have never run prometheus in production and it look more complicated to us ;-) )

but now I may have missed the purpose of the alarms you are refering too into influxDB.

@WadeBarnes can you shed some light here ? TIA

@GuillaumeBourque-QC
Copy link
Contributor

Hello @WadeBarnes for this project which is not too complex, I don't see the value to store all metrics in prometheus and influxDB.

I would put prometheus and alertmanager as optional since I was able to convert all dashboards from prometheus to InluxDB only and all alerts to granafa alerting. This will be push as a PR when my code clean up will be done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants