How to filter out blank lines and specific entries from log4net files

[marketier]

My Windows application is using the log4net library to generate log files. The logs contain blank lines and entries with the word "Setting:" along with the DEBUG entries. Here's a sample of the logs:

2022-07-12 12:12:20,615 [5928] DEBUG VehicleScore Calculate Driving Range - Start.
2022-07-12 12:12:20,615 [5928] DEBUG VehicleScore Updated By: 0
Setting: FlagToReset

2022-07-12 12:12:20,616 [5928] DEBUG VehicleScore Calculate Calculate Driving Range  - Start.

Can I use the Filter transform to exclude blank lines and entries containing the word "Setting:" ?

[jszwedko]

Hi @marketier !

You can use the filter transform to filter out lines based on defined conditions.

These also seem like they might be multiline logs? If so, you could also look at the multiline configuration for the file source to read the multiline log:

2022-07-12 12:12:20,615 [5928] DEBUG VehicleScore Updated By: 0
Setting: FlagToReset

As one event. Assuming that I'm correct that these are multiline logs and not just unformatted log lines.

[marketier]

Hi @jszwedko ,

Annoyingly, the logs are actually a combination of structured and unformatted entries. Thinking about it, I'm actually looking to remove any entry that does not start with a datetime.

Based on the above sample, lines 1, 2 and 5 are valid. The line beginning with "setting" and blank line needs to be excluded.

[jszwedko]

Hi @marketier ,

Gotcha, that should be relatively simple using a filter transform then. Something like:

[transforms.drop_invalid_lines]
type = "filter"
inputs = ["..."]
condition = """
matches!(.message, r'\d{4}-\d{2}-\d{2}.+')
"""

Which would drop any line that doesn't match the provided regular expression.

[marketier]

Hi @jszwedko ,

Do you know why this...

[transforms.filter_out_db]
type = "filter"
inputs = ["app_logs"]
condition = '!contains(.message, "DatabaseConnectionString")'

...would yield the following error?

2022-07-20T12:32:05.043631Z ERROR vector::topology: Configuration error. error=Transform "filter_out_db":
�[0m�[1m�[38;5;9merror[E100]�[0m�[1m: unhandled error�[0m
  �[0m�[36m┌─�[0m :1:1
  �[0m�[36m│�[0m
�[0m�[36m1�[0m �[0m�[36m│�[0m �[0m�[31mcontains(.message, "DatabaseConnectionString")�[0m
  �[0m�[36m│�[0m �[0m�[31m^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
  �[0m�[36m│�[0m �[0m�[31m│�[0m
  �[0m�[36m│�[0m �[0m�[31mexpression can result in runtime error�[0m
  �[0m�[36m│�[0m �[0m�[36mhandle the error case to ensure runtime success�[0m
  �[0m�[36m│�[0m
  �[0m�[36m=�[0m see documentation about error handling at https://errors.vrl.dev/#handling
  �[0m�[36m=�[0m learn more about error code 100 at https://errors.vrl.dev/100
  �[0m�[36m=�[0m see language documentation at https://vrl.dev

I am looking to exclude log entries containing the string "DatabaseConnectionString".

Thanks.

[spencergilbert]

.message may not be a String - and VRL is guiding you to ensure that, because contains requires the input value to be a String. The program below should handle the error case.

!contains(string!(.message), "DatabaseConnectionString")