Add /health page

[garethrees]

Consider signs that the app is having issues.

I expect we want to check that:

• There are no delivered submissions with empty references
• There are no submissions that have been unqueued for ages
• There are no submissions that have been queued for ages
• There's no old data that should have been cleaned up by the to-be-written cleanup task
• There are no jobs in the Sidekiq "Dead Jobs" queue

We can then add this to the test_urls vhost config.

[sagepe]

These data could also be exposed via the Prometheus endpoint (#56), although an aggregated set of data from across the service would be useful for service-wide metrics. If you do expose service-wide metrics at /health consider doing so using Prometheus's data format (there would be nothing to stop us parsing this inside a Nagios check, too).

[garethrees]

Example with Prometheus formatting https://api.mapumental.com/health/metrics

[sagepe]

Might be worth a look: https://github.com/Strech/sidekiq-prometheus-exporter

[garethrees]

Also want a check for "old" queued submissions with no reference where there are no jobs scheduled (or retry jobs scheduled) to guard against #89.