Rebooting GitLab may trigger ClamAV alarm

[dsotirho-ucsc]

The azul-clamscan-<deployment> alarm is triggered if a clamscan succeeded log message is not produced within an 18 hour period. Since the ClamAV scan is run twice daily, and takes many hours to complete, it is possible for a reboot of the GitLab instance (due to an update, backup, or testing) to cancel an ongoing ClamAV scan and prevent a successful completion of a scan to fall within an 18 hour window since one last completed.

Recommended solution is to increase the alarm's period to 24 hours.

Note: 24 hours is the maximum allowed time period for an alarm with one evaluation period. (from: Common features of CloudWatch alarms)

The number of evaluation periods for an alarm multiplied by the length of each evaluation period can't exceed one day.

[dsotirho-ucsc]

Assignee to provide symptoms and solution in description.

[hannes-ucsc]

For demo, reboot a GL instance on day before demo while scan is ongoing (prepare proof). Show that alarm did not go off.

I don't think we need an elaborate demo. We discovered that the attempted fixes from the first two PRs (#6155 and #6315) weren't effective before we even got to the demo. IOW, we will likely make the same discovery about PR #6374 organically, during normal operations.

[dsotirho-ucsc]

@hannes-ucsc: "Rebooting the instance still results in a false alarm, for example:"

https://groups.google.com/a/ucsc.edu/g/azul-group/c/zkmzRVv_rec/m/WbNWhMsKBQAJ

You are receiving this email because your Amazon CloudWatch Alarm "azul-clamscan-dev.alarm" in the US East (N. Virginia) region has entered the ALARM state, because "Threshold Crossed: 1 out of the last 1 datapoints [0.0 (26/04/24 09:48:00)] was less than the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition)." at "Saturday 27 April, 2024 09:48:13 UTC".

View this alarm in the AWS Management Console:
https://us-east-1.console.aws.amazon.com/cloudwatch/deeplink.js?region=us-east-1#alarmsV2:alarm/azul-clamscan-dev.alarm

Alarm Details:

• Name: azul-clamscan-dev.alarm
• Description:
• State Change: OK -> ALARM
• Reason for State Change: Threshold Crossed: 1 out of the last 1 datapoints [0.0 (26/04/24 09:48:00)] was less than the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition).
• Timestamp: Saturday 27 April, 2024 09:48:13 UTC
• AWS Account: 122796619775
• Alarm Arn: arn:aws:cloudwatch:us-east-1:122796619775:alarm:azul-clamscan-dev.alarm

Threshold:

• The alarm is in the ALARM state when the metric is LessThanThreshold 1.0 for at least 1 of the last 1 period(s) of 86400 seconds.

Monitored Metrics:

• MetricExpression: FILL(log_count_raw, 0)
• MetricLabel: No Label

State Change Actions:

• OK: [arn:aws:sns:us-east-1:122796619775:azul-monitoring-dev]
• ALARM: [arn:aws:sns:us-east-1:122796619775:azul-monitoring-dev]
• INSUFFICIENT_DATA:

[dsotirho-ucsc]

Assignee to consider increasing the frequency of the cronjob to */18 hours

[dsotirho-ucsc]

There is a contradiction in the above comment: */18 would not be an increase. Assignee to formalize plan.

[dsotirho-ucsc]

The alarm fires when a successful clamscan message wasn't logged within the last 24 hours. On average, a successful scan takes anywhere from 10 - 14 hours ( quicker on anvildev and prod, slower on anvilprod and dev).

Currently clamscan is set up to run twice a day. This causes the alarm to fire if the scan following a reboot takes longer than the scan that completed just prior to the reboot.

For this example, assume a 11 hour scan starting at 00 and 12:

start   00:00
end     11:00
start   12:00
reboot  13:00
start   00:00
end     11:05 (alarm fired at 11:01)

Since systemd timers won't start a service that is still running from its last activation by a timer, I purpose setting the clamscan's timer to run 6 times a day, or every 4 hours (*-*-* */4:00:00).

Example, 11 hour scan starting every 4 hours (00, 04, 08, 12, 16, 20):

start   00:00
end     11:00 *
start   12:00
reboot  13:00
start   16:00
end     03:00 *

11:00 to 03:00 = 16 hours

Example, 14 hour scan starting every 4 hours (00, 04, 08, 12, 16, 20):

start   00:00
end     14:00 *
start   16:00
reboot  17:00
start   20:00
end     10:00 *

14:00 to 10:00 = 20 hours

[dsotirho-ucsc]

@hannes-ucsc: "Let's just start the unit every hour. If the scan takes less than an hour, it's actually desirable to start it on the next full hour. Extra care to be taken to ensure that the scans aren't running in parallel or overlap."

[hannes-ucsc]

Note my edits to the demo instructions.

[dsotirho-ucsc]

Resolution was incomplete, recently a scan took 26 hours to complete following a reboot of GitLab dev.

CloudWatch logs:

[dsotirho-ucsc]

Spike to compare hierarchical folder sizes on the two lower GitLab instances using a treemap based GUI.

[achave11-ucsc]

Unfortunately, it isn't possible to generate a hierarchical folder size comparison of a given instance without proprietary software. Like TreeMap which features the ability to connect to the instance, perform the scan, and generate the desired map.
Most other options with the TreeMap type of GUI are similar in the sense that they 1) only run as a desktop apps, and/or 2) are unable to generate a report from/to a specified file and 3) are proprietary.

The viable open source choice was ncdu, which was mentioned multiple times in the provided search results from the treemap based GUI. It provides a text based interface which one may navigate to manually traverse the produced report.

dev_ncdu.json
anvildev_ncdu.json

To display the generated reports, install ncdu (via brew in MacOS), then run:

ncdu -f dev_ncdu.json  # Or anvildev_ncdu.json

[achave11-ucsc]

Originally, this was ran without sudo privileges, causing the inaccuracy observed during PL where the mnt directory didn't encompass the majority of the space usage.

The file download links in the previous comment have been updated to link to the ncdu report which used sudo privileges.