[draft] System logging consolidation (general observability)

[robertek]

Systemd has one real killer feature and it is to log everything that is executed under its hands and then be able to dig that info out conveniently using journalctl. This is something that dinit lack the most from my point of view. A robust system should log everything, regardless its importance. The first glance on many outputs may be that it is not worth logging, but this is true until it is needed to debug something (which is never known in advance).

So I wanted to open this placeholder/design/ideas issue (if there is a better place to put these kind of longer term projects let me know).
Referencing for example the #2657

I have done a quick review of dinit, various sysloging facilities and systemd and its journaling.

At the beginning I see a couple of issues that needs to be considered and then implemented or set-up.

1. Dinit does not have any global logging override, everything depends on the service.
   There are a couple of options.
   1.1. Every possible service would have a consistent output logging definition.
   1.2. Dinit would be patched to be able to log by default and possibly override the service logging definition.
   1.3. Special logging service as a logging collector and handler. This would still need a Dinit patching (to support that, buffer until started, etc), but it would have a couple of benefits. Dinit does not need to have a global configuration, the service may be restarted to rotate the logs, a dependency to a log directly mount target, etc.

2. Consistent logging format and structure.
   2.1. Systemd uses a custom made binary format, which may not be that easy to mimic and maintain long term (libsystemd dependency would solve this). But it provides journalctl as a known convenient tool and filter.
   2.2. There is a couple of standardized formats out there, that are text based. Probably the most suitable RFC 5424
   2.3. Any message should have at least a timestamp and service reference. Something may be encoded on a filesystem level (log file per service?) or a tag level (binary metadata, APP-NAME/PROCID RFC5424 tag, or elsewhere).

3. A log rotation/retention mechanism.
   3.1. Different according to previous decision.
   3.2. In case of text logs, a boot based rotation would be probably enough at the beginning (with some size caping).
   3.3. An interesting idea: In case of separate logging directory on ZFS a possible use of snapshots for rotation.

4. A log querying and searching mechanism.
   4.1. Journalctl does that already but needs systemd internal binary format.
   4.2. RFC5424 has a broad support for external tools and may be conveniently handled by syslog-ng.
   4.3. Both RFC5424 and systemd journal (through libsystemd) has support in C, Python, Rust or other to implement custom querying tool in.
   4.4. RFC5424 may be filtered also using regular shell tools.
   4.5. It should be possible at least to filter messages according to service, time range, boot (current/previous boot)
   4.6. This decision is affected (or may affect) the choice of logfile filesystem structure (file per service, single log file, etc).

Non-service logging output may be logged using a helper service (kmesg, etc).

A more distant and complex task is around graphical apps/services logging. For example Plasma utilize systemd units generation to execute any service/app (depends on #3051). But this is out of scope now.