Fluent::Plugin::Anomalydetect, a plugin for Fluentd

To detect anomaly for log stream, use this plugin. Then you can find changes in logs casually.

Installation

Add this line to your application's Gemfile:

gem 'fluent-plugin-anomalydetect'

And then execute:

$ bundle

Or install it yourself as:

$ gem install fluent-plugin-anomalydetect

Usage

<source>
  type file
  ...
  tag access.log
</source>

<match access.**>
  type anomalydetect
  tag anomaly.access
  tick 86400
</match>

<match anomaly.access>
  type file
  ...
</match>

Then the plugin output anomaly log counts in each day.

This plugin watches a value of input record number in the interval set with tick.

If you want to watch a value for a target field in data, write below:

<match access.**>
  type anomalydetect
  tag anomaly.access
  tick 86400
  target fieldname
</match>

more configuration

<match access.**>
  type anomalydetect
  tag anomaly.access
  tick 86400
  target fieldname
  outlier_term 7
  outlier_discount 0.5
  smooth_term 7
  score_term 28
  score_discount 0.01
</match>

If you want to know detail of these parameters, see "Theory".

<match access.**>
  type anomalydetect
  ...
  store_file /path/to/anomalydetect.dat
</match>

If "store_file" option was specified, a historical stat will be stored to the file at shutdown, and it will be restored on started.

<match access.**>
  type anomalydetect
  ...
  threshold 3
</match>

If "threshold" option was specified, plugin only ouput when the anomalyscore is more than threshold.

<match access.**>
  type anomalydetect
  ...
  trend up
</match>

If "trend" option was specified, plugin only ouput when the input data tends to up (or down).

Parameters

• outlier_term

• outlier_discount

• smooth_term

• score_term

• score_discount

• tick

  The time interval to watch in seconds.

• tag

  The output tag name. Required for aggregate all. Default is anomaly.

• add_tag_prefix

  Add tag prefix for output message. Required for aggregate tag.

• remove_tag_prefix

  Remove tag prefix for output message.

• aggragate

  Process data for each tag or all. The default is all.

• target

  Watch a value of a target field in data. If not specified, the number of records is watched (default). The output would become like:

    {"outlier":1.783,"score":4.092,"target":10}
  

• threshold

  Emit message only if the score is greater than the threshold. Default is -1.0.

• trend

  Emit message only if the input data trend is up (or down). Default is nil.

• store_file

  Store the learning results into a file, and reload it on restarting.

• targets

  Watch target fields in data. Specify by comma separated value like x,y. The output messsages would be like:

    {"x_outlier":1.783,"x_score":4.092,"x":10,"y_outlier":2.310,"y_score":3.982,"y":3}
  

• thresholds

  Threahold values for each target. Specify by comma separated value like 1.0,2.0. Use with targets option.

• outlier_suffix

  Change the suffix of emitted messages of targets option. Default is _outlier.

• score_suffix

  Change the suffix of emitted messages of targets option. Default is _score.

• target_suffix

  Change the suffix of emitted messages of targets option. Default is `` (empty).

• suppress_tick

  Suppress to emit output messsages during specified seconds after starting up.

Theory

データマイニングによる異常検知

ToDo

FFT algorithms

Copyright

• Copyright

  • Copyright (c) 2013- Muddy Dixon
  • Copyright (c) 2013- Naotoshi Seo

• License

  • Apache License, Version 2.0