Add a diagnostic refresh_age metric and corresponding container healthcheck

[michaelosthege]

Hi @sfudeus, thanks for this great exporter!

Every now and then my container appears to hang, producing such gaps until I restart it:

I assume that some internal loop crashes, but the logs don't indicate anything.

To notice when things to out of order, I introduced a new metric:

# HELP homematic_refresh_age Seconds since the last successful refresh.
# TYPE homematic_refresh_age gauge
homematic_refresh_age{ccu="192.168.178.30"} 4.156044244766235

In combination with a healthcheck script and docker-autoheal this makes it easy to automatically restart the container.

[sfudeus]

Hi @michaelosthege, thanks for the contribution. This looks good besides the one comment I had. Ultimately, instead of needing such a healthcheck long-term, I'd prefer finding and preventing an issue which would block the loop infinitely. But at least this helps finding such occasions.

[michaelosthege]

I'd prefer finding and preventing an issue which would block the loop infinitely. But at least this helps finding such occasions.

Totally agree. I tried to debug, but found it quite hard.
Mainly two things prevented me from proper debugging:

• The DEBUG mode is too verbose to be useful for debugging rare events
• The generate_metrics is quite complex, with lots of branches and a lots of indentation.

As a start I would recommend to apply ruff and refactor some things into functions.
Let me know if I shall open a PR!

[sfudeus]

Mainly two things prevented me from proper debugging:

• The DEBUG mode is too verbose to be useful for debugging rare events
• The generate_metrics is quite complex, with lots of branches and a lots of indentation.

As a start I would recommend to apply ruff and refactor some things into functions. Let me know if I shall open a PR!

Feel free of you find the time for it. The exporter has grown over time and could need some rework.
Please try to break the refactoring down into reasonable chunks to not have one complex big refactoring which is hard to review.

[sfudeus]

@michaelosthege Can you rebase on the latest state of master? I recently refactored the workflow definitions and the image build job fails because of that mismatch.

[michaelosthege]

Done!

[sfudeus]

Workflow security setting prevent running from a remote repo, so I cloned it into a local branch on PR #58.
Image preview available at docker.io/sfudeus/homematic_exporter:preview-58

[sfudeus]

no curl in the docker image yet - and the scripts lacks an execute bit.

[michaelosthege]

Oh wow, I should have checked the healthcheck logs!

I didn't need to change the file permissions though. It's now doing 0-exit on my deployment..

[sfudeus]

your healthcheck command included the bash-invocation and only gave the script as argument, that is why you didn't nee the bit. But it would make sense to do so to be able to call it natively.

[michaelosthege]

Please feel free to suggest a change or make a commit. I don't know what exactly you'd like to see here

[sfudeus]

See 5f580ed, it is just about the permissions bits on the healthcheck.sh. I can add that later myself as well, I cannot push to your branch.

The most recent docker build contains that already and looks good so far in my cluster. I did not set a liveness check using the healthcheck yet.

[michaelosthege]

The Linux file permissions at the git level? I didn't know these were copied into the docker image.