Support structured logback messages via logstash encoder

[trask]

This has come up a few times:

• Consider adding a logs API or allowing languages to present a logs API within the OpenTelemetry API opentelemetry-specification#2234 (comment)
• Send logs to OTel collector with logback (using OpenTelemetryAppender) #5496
• Structured logging not working microsoft/ApplicationInsights-Java#2252

Unfortunately, logback has no native structured log message support (other than MDC, which is not ideal, see discussion in first link above).

Logstash Json Encoder seems to be reasonably popular and fills this gap. Unfortunately, the logstash StructuredArgument API is defined around arbitrary Json data, and so maybe(?) the best we can do from translating it to OTel attributes is to take the top-level key/values, where the values may be arbitrarily nested json strings themselves.

[trask]

Unfortunately, logback has no native structured log message support (other than MDC, which is not ideal, see discussion in first link above).

btw, slf4j2 now has structured log message support, so this may be less important now

[oldium]

Just faced similar issue when migrating from Logstash to OpenTelemetry auto-instrumentation. The code uses LogstashMarker API (actually ObjectAppendingMarker instances, which implements StructuredArgument API) to add some key-value pairs to the log output. And if I read the code correctly, this is transformed into JSON by ArgumentsJsonProvider.

It would be good if OTEL is able to transform StructuredArguments into LogRecord Attributes. For primitive values it could be simple key-value pair, for arrays or object values the value could be JSON serialization into a string.