Filter Logs Before Remap

[jamalrahaman]

I am trying to eliminate some logs even before remap in the Transforms section.
i am using File as my source.
i tried datadog_search but i am able to do conditioning only once.
how can i do multi conditioning in filters?

Below example is for Datadog_search but i am happy to use VRL too. Suggest some ideas

[transforms.reduce_logs]
type = "filter"
inputs = [ "logs" ]
condition = { type = "datadog_search", source = "PASSIVE SERVICE CHECK" }
#condition = '.message != "PROCESS_SERVICE_CHECK_RESULT"'

[transforms.modify]
type = "remap"
inputs = ["reduce_logs"]
source = '''
. = parse_groks!(
.message,
patterns: [
"\[%{NUMBER:nagios_epoch}\] Warning:%{SPACE}%{GREEDYDATA:nagios_message}",
"\[%{NUMBER:nagios_epoch}\] %{NAGIOS_TYPE_CURRENT_SERVICE_STATE:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statetype};%{DATA:nagios_statecode};%{GREEDYDATA:nagios_message}
'''
)

TIA

[spencergilbert]

You could chain together multiple filtering components before remap if you wanted, or use the OR expression (||) to chain together multiple VRL expressions in a single filter.

[jamalrahaman]

I am not able to filter logs directly from sources.

for the below code it is not taking any logs from stdin.

if you can share me some sample filters to filter from source it would be helpful

[sources.logs]
type ="stdin"

[transforms.reduce_logs]
type = "filter"
inputs = [ "logs" ]
#condition = { type = "datadog_search", source = "PASSIVE SERVICE CHECK"}
condition = 'source == "PROCESS_SERVICE_CHECK_RESULT"'

[spencergilbert]

data_dir = "/tmp/vector/12274"

[sources.source0]
type = "stdin"

[sources.source0.decoding]
codec = "json"

[transforms.transform0]
inputs = ["source0"]
type = "filter"
condition = '''
.source == "PROCESS_SERVICE_CHECK_RESULT"
'''

[sinks.sink0]
inputs = ["transform0"]
target = "stdout"
type = "console"

[sinks.sink0.encoding]
codec = "json"

❯ vector -c vector.toml
2022-04-19T12:42:29.769088Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,kube=info"
2022-04-19T12:42:29.769173Z  INFO vector::app: Loading configs. paths=["vector.toml"]
2022-04-19T12:42:29.770745Z  INFO vector::sources::stdin: Capturing STDIN.
2022-04-19T12:42:29.773101Z  INFO vector::topology::running: Running healthchecks.
2022-04-19T12:42:29.773145Z  INFO vector::topology::builder: Healthcheck: Passed.
2022-04-19T12:42:29.773241Z  INFO vector: Vector has started. debug="false" version="0.21.0" arch="x86_64" build_id="c1edb89 2022-04-14"
2022-04-19T12:42:29.773397Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
{"source": "PROCESS_SERVICE_CHECK_RESULT", "message": "foobar", "id": 42}
{"host":"COMP-J44M27KKWV","id":42,"message":"foobar","source":"PROCESS_SERVICE_CHECK_RESULT","source_type":"stdin","timestamp":"2022-04-19T12:42:48.717542Z"}
{"source": "ANOTHER_SOURCE", "message": "filtered out", "id": 0}

condition = 'source == "PROCESS_SERVICE_CHECK_RESULT"'

Your use of source is an undefined variable, not a path/field on an event - the filter will never resolve to true and pass no events. You can find documentation on path syntax here.

[jamalrahaman]

my source is not a Json

echo "[1649679264] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;stage-kaleidoscope-platform-check;stag-kaleidos-resolveIncidents-lambda-errors-alert;0;Threshold Crossed: 1 out of the last 10 datapoints [0.0 (11/04/22 12:12:00)] was not greater than or equal to the threshold (10.0) (minimum 6 datapoints for ALARM -> OK transition)." | sudo vector --config-toml vector.toml

what must be my Filter to filter out from the source now?

[sources.source0]

type = "stdin"



[sources.source0.decoding]

codec = "json"



[transforms.transform0]

inputs = ["source0"]

type = "filter"

condition = '''

.message == "PROCESS_SERVICE_CHECK_RESULT"

'''



[sinks.sink0]

inputs = ["transform0"]

target = "stdout"

type = "console"



[sinks.sink0.encoding]

codec = "json"

[monitor@op5maws-va-b001p jamal]$ echo "[1649679264] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;stage-kaleidoscope-platform-check;stag-kaleidos-resolveIncidents-lambda-errors-alert;0;Threshold Crossed: 1 out of the last 10 datapoints [0.0 (11/04/22 12:12:00)] was not greater than or equal to the threshold (10.0) (minimum 6 datapoints for ALARM -> OK transition)." | sudo vector --config-toml vector.toml
2022-04-19T13:50:15.983374Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info"
2022-04-19T13:50:15.983459Z  INFO vector::app: Loading configs. paths=["comcast.toml"]
2022-04-19T13:50:15.986982Z  INFO vector::sources::stdin: Capturing STDIN.
2022-04-19T13:50:15.987415Z  INFO vector::topology::running: Running healthchecks.
2022-04-19T13:50:15.987499Z  INFO vector::topology::builder: Healthcheck: Passed.
2022-04-19T13:50:15.987513Z  INFO vector::topology::running: Starting source. key=source0
2022-04-19T13:50:15.987558Z  INFO vector::topology::running: Starting transform. key=transform0
2022-04-19T13:50:15.987580Z  INFO vector::topology::running: Starting sink. key=sink0
2022-04-19T13:50:15.987632Z  INFO vector: Vector has started. debug="false" version="0.20.1" arch="x86_64" build_id="38b806e 2022-04-07"
2022-04-19T13:50:15.987644Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2022-04-19T13:50:15.987736Z  WARN source{component_kind="source" component_id=source0 component_type=stdin component_name=source0}: vector::internal_events::decoder: Failed deserializing frame. error=Error parsing JSON: Error("trailing characters", line: 1, column: 14) internal_log_rate_secs=10
2022-04-19T13:50:15.987809Z  INFO source{component_kind="source" component_id=source0 component_type=stdin component_name=source0}: vector::sources::stdin: Finished sending.
2022-04-19T13:50:15.987811Z  INFO vector::shutdown: All sources have finished.
2022-04-19T13:50:15.987825Z  INFO vector: Vector has stopped.

[spencergilbert]

You passed an event into the source, the event is created as .message = "[1649679264] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;stage-kaleidoscope-platform-check;stag-kaleidos-resolveIncidents-lambda-errors-alert;0;Threshold Crossed: 1 out of the last 10 datapoints [0.0 (11/04/22 12:12:00)] was not greater than or equal to the threshold (10.0) (minimum 6 datapoints for ALARM -> OK transition)."

Your filter is checking equality of that .message filed with "PROCESS_SERVICE_CHECK_RESULT", which does not equal. Either parse your .message into a structured format that allows you to compare exactly to the field you want or use a more advanced function, ex: contains, match.

[jamalrahaman]

Thanks @spencergilbert . One last Question from my side . Is it possible to convert Epoch time to Time stamp ?

[jszwedko]

Hi @jamalrahaman ! You can use to_timestamp() in VRL to convert an epoch time to a native timestamp. See https://vector.dev/docs/reference/vrl/functions/#to_timestamp

[jamalrahaman]

Woww ! @jszwedko and @spencergilbert Thanks a lot for your support ! i was able to achieve what i wanted :)