ELK Connector for Node Application Metrics

A connector that collects data using 'appmetrics' and sends it to a configured ElasticSearch instance in LogStash format for use with Kibana.

Getting Started

Installation

The ELK Connector for Node Application Metrics can be installed via npm:

$ npm install appmetrics-elk

This is designed to be used with an ElasticSearch database, and a visualization tool such as Kibana.

Configuring the ELK Connector for Node Application Metrics

The connector can be used in your application by requiring it as the first line of your application:

var appmetrics = require('appmetrics-elk').monitor();

This will send all of the available appmetrics data to the ElasticSearch instance, as well as returning an appmetrics object that can be used to control data collection.

var appmetrics = require('appmetrics-elk').monitor();
appmetrics.disable('mysql');            // disable MySQL monitoring

Additionally, the monitor() API call can be passed an optional ElasticSearch Configuration object to configure the ElasticSearch connection, including database location and security.

The same configuration object can be used to pass configuration to the ELK connector. The following configurations can be applied:

• index (String) the name of the index to use for storing the monitoring data. The default is appmetrics.
• applicationName (String) the name to use for the applicationName field in the monitoring data. The default is the name of the applications main file, eg. app.js.

var config = {
    hosts: [
        'https://es1.bluemix.net',
        'https://es2.bluemix.net'
    ],
    ssl: {
        ca: fs.readFileSync('./cacert.pem'),
        rejectUnauthorized: true
    },
    index: 'nodedata',
    applicationName: 'HelloWorld'
}

var appmetrics = require('appmetrics-elk').monitor(config);
appmetrics.disable('mysql');            // disable MySQL monitoring

Data Provided to ElasticSearch

The ELK Connector for Node Application Metrics uploads its data to the 'appmetrics' index in ElasticSearch. It sends the following values to ElasticSearch for every monitoring entry:

Value	Description
timestamp	The time when the monitoring event occurred
hostName	The hostname for the machine the monitored process is running on
pid	The process ID for the monitored process
applicationName	The JavaScript file used to launch the application, or a custom name

Additional data is then included depending on the monitoring event.

CPU Data

Value	Description
cpu.process	The CPU usage of the application as a percentage of total machine CPU
cpu.system	The CPU usage of the system as a percentage of total machine CPU

Memory Data

Value	Description
memory.process.private	The memory used by the application that cannot be shared with other processes, in bytes
memory.process.physical	The RAM used by the application in bytes
memory.process.virtual	The memory address space used by the application in bytes
memory.system.physical	The total amount of RAM in use on the system in bytes
memory.system.total	The total amount of RAM available on the system in bytes

Garbage Collection Data

Value	Description
gc.used	The JavaScript heap used by the application in bytes
gc.size	The size of the JavaScript heap in bytes
gc.type	The type of GC cycle, either 'M' or 'S'
gc.duration	The duration of the GC cycle in milliseconds

Event Loop Latency Data

Value	Description
eventloop.latency.min	The shortest sampled latency for processing an event
eventloop.latency.max	The longest sampled latency for processing an event
eventloop.latency.avg	The mean sampled latency for processing an event

HTTP Request Data

Value	Description
http.method	The HTTP method used for the request
http.url	The URL on which the request was made
http.duration	The time taken for the HTTP request to be responded to in ms

Socket.io Data

Value	Description
socketio.method	The type of socketio event, either broadcast, emit or receive
socketio.event	The name of the socketio event
sokcetio.duration	The time taken for the socketio event to be handled in ms

MongoDB Query Data

Value	Description
mongo.query	The query made of the MongoDB database
mongo.duration	The time taken for the MongoDB query to be responded to in ms

MySQL Query Data

Value	Description
mysql.query	The query made of the MySQL database
mysql.duration	The time taken for the MySQL query to be responded to in ms

PostgreSQL Query Data

Value	Description
postgres.query	The query made of the PostgreSQL database
postgres.duration	The time taken for the postgreSQL query to be responded to in ms

Redis Command Data

Value	Description
redis.cmd	The Redis command sent to the server or 'batch.exec'/'multi.exec'
redis.duration	The time taken for the Redis command to be handled to in ms

Memcached Operation Data

Value	Description
memcached.method	The method used in the memcached client, eg set, get, append, delete, etc.
memcached.key	The the key associated with the data
memcached.duration	The time taken for the memcached command to be handled to in ms

Leveldown Query Data

Value	Description
leveldown.method	The leveldown method being used.
leveldown.key	The key being used for a call to get, put or del
leveldown.value	The value being added to the LevelDB database using the put method
leveldown.opCount	The number of operations carried out by a batch method
leveldown.duration	The time taken for the leveldown query to be handled to in ms

MQTT Message Data

Value	Description
mqtt.method	Either a publish or a received message
mqtt.topic	The topic on which the message is published or received.
mqtt.qos	The quality of service used for the message
mqtt.duration	The time taken for the message to be handled to in ms

MQLight Message Data

Value	Description
mqlight.method	Either a send or a received message
mqlight.topic	The topic on which the message is published or received.
mqlight.qos	The quality of service used for the message
mqlight.data	The first 25 characters of the message
mqlight.duration	The time taken for the message to be handled to in ms

Sending Custom Data to ElasticSearch

The Node Application Metrics to ELK Connector registers for events that it is aware of, and forwards the data from those events to ElasticSearch. The registration for those events is based on the 'mappings' files in the following directory:

node_modules/appmetrics-elk/mappings/

Any mappings files found in that directory are both used to configure how ElasticSearch handles the data, and to configure the monitoring events that are forwarded.

The type field is used to determine the name of the event to register for, and the properties fields are used to determine the values to send. Note that the values in the properties entry in the mapping file must match the fields in the monitoring event data. For example, the CPU event has the following data:

• process
• system

The mapping file that causes this data to be sent to ElasticSearch therefore has the following structure:

{
    "index":  "appmetrics",
    "type":   "cpu",
    "body": {
        "_source" : {"compress" : true},
        "_ttl" : {"enabled" : true, "default" : "90d"},
        "properties": {
            "timestamp":    {"type": "date", "format": "dateOptionalTime"},
            "hostName":     {"type": "string", "index": "not_analyzed"},
            "pid":          {"type": "integer"},
            "cpu": {
                "type": "nested",
                "include_in_parent": true,
                "properties": {            
                    "process":      {"type": "float"},
                    "system":       {"type": "float"}
                }
            }
        }   
    }
}

This causes the Node Application Metrics to ELK Connector to register for cpu events and forward the process and system values as cpu.process and cpu.system.

Using Kibana with the ELK Connector for Node Application Metrics

During startup the ELK Connector for Node Application Metrics attempts to provide some pre-configuration for using Kibana 4 with the provided data. It does this by uploading the following if there are not existing ones already associated with the 'appmetrics' index:

• An index pattern
• Data mappings for the data types
• Default visualizations for the data types
• A default dashboard

Each of these configurations are dynamically loaded from the 'indexes', 'mappings', 'charts' and 'dashboards' directories in the appmetrics-elk install directory. It is therefore possible to prevent the configurations from being automatically added by deleting those files, or to add to them by adding existing files.

Note: The 'mappings' directory also provides the configuration of which types of monitoring data are uploaded to ElasticSearch so entries should only be deleted if necessary. See Sending Custom Data to ElasticSeach for more information.

Visualizing the data with Kibana 4

The pre-configuration for Kibana 4 provdes a number of default visualizations, as well as a default dashboard. These can subsequently be modified or new visualizations and dashboards created.

Using the dashboard
In order to avoid replacing any dashboard already in use in Kibana 4 with the one supplied by the Node Application Metrics to ELK Connector, the dashboard is made available but not loaded. The dashboard is loaded by:

1. Click on the "Dashboard" tab
2. Select the "Load Saved Dashboard" icon
3. Select "Default AppMetrics Dashboard" from the list of saved dashboards
   This now loads a simple dashboard that uses some of the default visualization charts provided by the Node Application Metrics to ELK Connector.

Using the visualization charts
In addition to the dashboard, a number of visualization charts are also provided. These can be added to a dashboard using the following steps:

1. Click on the "Dashboard" tab
2. Click on the "Add Visualization" icon
3. Select a visualization chart from the menu
4. Place and resize the visualization chart by dragging it across the screen

You can also create your own charts using the "Visualize" tab.

License

The Node Application Metrics to ELK Connector is licensed using an Apache v2.0 License.

Version

The current version is 1.0.2

1.0.2 Support for new monitoring data
1.0.1 Support for configurable indexes and addition of applicationName field
1.0.0 Initial release