Monitoring

Overview

The service allows for monitoring its metrics using the following means:

• JMX
• using HTTP Management API

Management API

Management API exposes the metrics endpoint which returns metrics information about the service and its components:

• caches statistics
• memory (free and used memory)
• threads (number started and peak number of threads)
• garbage collector

For detailed description of available metrics please refer to Spring Boot documentation Metrics.

The metrics endpoint can be accessed using HTTP client, e.g. curl:

curl -X GET -u admin:secret http://localhost:13021/manage/metrics

Manage endpoints are protected using HTTP Basic Authentication, please refer to Configuration for details.

The endpoint returns a JSON structure which looks like this:

{
"mem": 364397,
"mem.free": 125572,
"processors": 4,
"instance.uptime": 68618,
"uptime": 78561,
"systemload.average": 0.72,
"heap.committed": 273408,
"heap.init": 65536,
"heap.used": 147664,
"heap": 524288,
"nonheap.committed": 92816,
"nonheap.init": 2496,
"nonheap.used": 90989,
"nonheap": 0,
"threads.peak": 61,
"threads.daemon": 12,
"threads.totalStarted": 71,
"threads": 61,
"classes": 12832,
"classes.loaded": 12832,
"classes.unloaded": 0,
"gc.g1_young_generation.count": 50,
"gc.g1_young_generation.time": 441,
"gc.g1_old_generation.count": 0,
"gc.g1_old_generation.time": 0,
"cache.commentstore.miss.ratio": 0.33333333333333337,
"cache.filecontentcache.size": 22,
"cache.gists.hit.ratio": 0.5,
"cache.metadatastore.size": 1,
"cache.metadatatore.size": 0,
"cache.metadatastore.miss.ratio": 0.1428571428571429,
"cache.gists.size": 1,
"cache.comments.size": 1,
"cache.filecontentcache.miss.ratio": 0,
"cache.historystore.size": 23,
"cache.metadatastore.hit.ratio": 0.8571428571428571,
"cache.historystore.hit.ratio": 0.125,
"cache.gists.miss.ratio": 0.5,
"cache.sessionkeys.miss.ratio": 0.33333333333333337,
"cache.historystore.miss.ratio": 0.875,
"cache.comments.hit.ratio": 0,
"cache.commentstore.hit.ratio": 0.6666666666666666,
"cache.sessionkeys.hit.ratio": 0.6666666666666666,
"cache.comments.miss.ratio": 1,
"cache.sessionkeys.size": 1,
"cache.commentstore.size": 1,
"cache.filecontentcache.hit.ratio": 1,
"gauge.response.gists.gistId": 1141,
"gauge.response.gists.gistId.comments": 49,
"counter.status.200.gists.gistId.comments": 1,
"counter.status.200.gists.gistId": 2
}

Collecting Metrics Over Time

To collect the metrics over time at specified intervals so it can be analysed, you can use the following script, which polls the metrics endpoint at specified interval and dumps the returned JSON to a directory:

#!/bin/bash

HOST=$(hostname)
PORT=13021
METRICS_ENDPOINT=/manage/metrics

URL=http://$HOST:$PORT/$METRICS_ENDPOINT


output_dir=/tmp/rcloud-gist-service-drop
interval=5

usage() { 
   echo "Usage: $0 -u <USER_NAME> -p <PASSWORD> [-o <OUTPUT_DIR>] [-i <INTERVAL>]" 1>&2; exit 1; 
}

while getopts ":u:p:o:i:" o; do
    case "${o}" in
        u)
            username=${OPTARG}
            ;;
        p)
            password=${OPTARG}
            ;;
        o)
            output_dir=${OPTARG}
            ;;
        i)
            interval=${OPTARG}
            ;;
        *)
            usage
            ;;
    esac
done

shift $((OPTIND-1))

if [ -z "${username}" ] || [ -z "${password}" ]; then
    usage
fi

if [ -e $output_dir ]; then
   echo "ERROR: Output directory already exist, please delete it first" 1>&2;
   usage
fi

if [ ! -e $output_dir ]; then
   mkdir -p $output_dir
fi

index=1
while true; do
  curl -u ${username}:${password} -H "Accepts: application/json" ${URL} -o ${output_dir}/${index}.json
  sleep $interval
  index=$(( $index + 1 ))
done 

JMX

If firewall settings allow for external JMX connections to the box where the service is exposed, it is possible to configure rcloud-gist-service JVM to accept JMX connections from remote hosts.

JMX configuration should be added to /opt/rcloud-gist-service/<JAR_BASE_NAME>.conf file, by extending JAVA_OPTS variable, e.g.:

JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=<HOST_NAME>"

Update <HOST_NAME> with the name of the host where the service is deployed to.

Note, the example JMX configuration turns off all JMX security, please refer to Monitoring and Management Using JMX Technology for details on securing JMX.

After (re-)starting the service you will be able to connect to it via JMX with jconsole or VisualVM using the url: <HOST_NAME>:8000.