Add a init/health check for mountpoints

[achimnol]

There are many cases in production setups that filesystem mounts fail or get disconnected at runtinme. Also some high-performance NFS backends take a very long time (e.g., 1-2 minutes) to initialize their mounts when the server is rebooted.

Although we are dealing with this kind of issues by adding a busy-wait loop in the runner scripts or mangling some systemd configurations, but it is sometimes difficult to ensure the customers that it's not our problems.

Detection

HOWTO

Let's use mountpoint to check/wait if the given set of directories are actually mounted or not upon startup.
If they fail to mount for a long time (e.g., 10 minutes), let's make agents and storage-proxy to actively fail and show appropriate error logs.

Mitigation

• Either:
  • Explicitly reject kernel creation requests with vfolder mounts on lost volumes with proper error messages.
  • Turn the agent's schedulable to off.
• Make an alarm to the cluster administrator.
  • ...so that he/she could escalate the issue to the infrastructure team/manager.
  • After the infra-level issue is resolved, we could instruct the administrator to turn on schedulable of the agent or restart the agent.

Steps

Preview Give feedback

1. Add status_history to agents to keep track of status changes and reasons #1843
   comp:agent comp:manager +2
   
2. Add a manager endpoint to publish monitoring alarm events #1845
   comp:manager +1
   
3. Integrate mountpoint health check with external monitoring systems like Zabbix (configurable) #1844
   area:infrastructure comp:agent +2
   
4. Perform periodic health checks of mountpoint by agents (configurable) #1846
   comp:agent +1
   
5. Check mountpoint readiness on startup with configurable retries and timeouts #1847
   comp:agent +1
6. Webhook support for monitoring alarms #104
   comp:agent comp:manager +2
7. Avoid assigning new sessions if free disk space of agent is below a threshold #3055
   area:infrastructure comp:agent effort:normal impact:invisible platform:enterprise type:feature urgency:3 +7

Expected result

This will greatly reduce the field support efforts that are related to storage issues.

[achimnol]

#104 would be also a part of this issue, sharing and extending the same proposed monitoring alarm subsystem. @HyeockJinKim

[achimnol]

Also a new consideration:

• When the systemcall hangs up inside the kernel, even mountpoint command goes into the deadlock (D) state. If this happens, we should not indefinitely repeat executing mountpoint to avoid accumulation of the D-state processes. We should apply a subprocess-level timeout to detect if this happens and makes an alarm.

[HyeockJinKim]

I'm thinking of working with the following structure, and I'd like to ask for your thoughts on it. @achimnol

1. perform a full mountpoint check with a timeout(like 5 sec) at a interval (like 5 min)
   • timeout and interval is configurable.
   • The periodic behavior is independent of the server
   • Registering and unregistering mount points in the inspector when mounting
2. return a CheckResult value for success/failure after checking
3. perform processing on CheckResult in AlertSystem (currently only logging)