As a mySociety employee, I want to be notified if an FOI request can't be added to Infreemation, so that I can fix the problem

[garethrees]

• Exception notifications (Set up exception notification #69)
• Queue health (Add /health page #76)
• Lock down /health

[garethrees]

@sagepe sorry, I didn't get to this. Is it okay if I leave it to you? Just need to lock down /health in the same way as /sidekiq for the production site.

[sagepe]

I've already prepared this. As it's not specific to this project, I've added it to the shared top-level template. It should be triggered by adding "prometheus_service_health": "/health/metrics" to the deployment JSON (see commit 27a6292b5 in the internal servers repo).

[sagepe]

Just noting that the prometheus_service_health key has been added to the staging site JSON ready for the next deploy.

[garethrees]

@sagepe do you consider this "done" now?

I can see that:

• We have metrics at /health/metrics that has locked down access
• We receive exception notification emails when there's a problem

Anything that you think we're missing?

[sagepe]

If I understand this ticket properly then I think it's covered for now, although we're not yet alerting directly on the metrics we're gathering just yet.

[garethrees]

Yeah, that's fine for now I think.

Could you update the wiki page with the basic flow of where these are going (I guess something like /health/metrics → collector process → ? → ?`) and how to view them over time?

[sagepe]

I've added some basic note to the FOIfC wiki page with a link to the outline of how the discovery works on the Prometheus page - do you want any more detail than this for now?

[garethrees]

Perfect, thanks! Its more just a note that "this exists and this is roughly what's happening". It can be expanded further if/when there are specific questions that aren't answered by the wiki.

And with that, this is done! 🚀