How can I pass hostname to log4j configuration?

[rkuccf]

Hi,
I'm working on log4j configuration that will send broker logs through SocketAppender to different pod. I want to set application appender's property to $HOSTNAME. I was trying to do this by setting additional java property using kafka.jvmOptions.javaSystemProperties and kafka.template.kafkaContainer.env:

---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
spec:
  kafka:
    jvmOptions:
      javaSystemProperties:
        - name: HOSTNAME
          value: $(HOSTNAME)
    logging:
      loggers:
        kafka.root.logger.level: INFO
        log4j.appender.CONSOLE: org.apache.log4j.ConsoleAppender
        log4j.appender.CONSOLE.layout: org.apache.log4j.PatternLayout
        log4j.appender.CONSOLE.layout.ConversionPattern: '%d{ISO8601} %p %c [%t] - %m%n'
        log4j.appender.SOCKET: org.apache.log4j.net.SocketAppender
        log4j.appender.SOCKET.Application: ${HOSTNAME}
        log4j.appender.SOCKET.Port: '8081'
        log4j.appender.SOCKET.RemoteHost: kafka-fluentbit-svc
        log4j.logger.kafka: INFO
        log4j.logger.kafka.controller: INFO
        log4j.logger.org.apache.kafka: INFO
        log4j.logger.state.change.logger: INFO
        log4j.rootLogger: ${kafka.root.logger.level}, CONSOLE, SOCKET
      type: inline
    template:
      kafkaContainer:
        env:
          - name: JAVA_TOOL_OPTIONS
            value: '-DHOSTNAME=$(HOSTNAME)'
    version: 3.7.0
  

Both of ways doesn't work and in the broker pod I can see:

[kafka@kafka-kafka-0 kafka]$ set | grep 'JAVA_TOOL_OPTIONS\|STRIMZI_JAVA_SYSTEM_PROPERTIES'
JAVA_TOOL_OPTIONS='-DHOSTNAME=$(HOSTNAME)'
STRIMZI_JAVA_SYSTEM_PROPERTIES='-DHOSTNAME=$(HOSTNAME)'

It seems that expanding environment variables doesn't work.
Is there any other way to pass hostname to log4j configuration?

Thank you in advance!

[mayndi15]

One solution would be to use the system properties lookup in Log4j (sys:HOSTNAME) and ensure that the HOSTNAME variable is correctly passed as a system property

[mayndi15]

Maybe something like

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
spec:
  kafka:
    template:
      kafkaContainer:
        env:
          - name: HOSTNAME
            valueFrom:
              fieldRef:
                fieldPath: spec.node
        command:
          - sh
          - -c
          - "/opt/kafka/set-hostname.sh" 

[scholzj]

You have to follow the CRD API and what it supports. You cannot make up things that don't exist.

TBH, I do not know if this is possible. Normally I would recommend to using the standard practices and collect the logs through the container engine and tools such as Fluentd etc.

[rkuccf]

Generally I would love to use Fluent if I could. The issue here is related to problem that my K8s vendor have with big amount of logs. They have multiple K8s clusters that are used by different teams and only one service that collects logs from each of these clusters. The service collects logs from container stdout. So to limit amount of logs I have to reduce this what is produce to stdout. I had two ideas:

• configure log4j to write logs to files and add additional sidecar with fluentbit the will read these files and send to elastic stack - unfortunately strimzi doesn't support any additional sidecars,
• configure log4j to use SocketAppender and implement additional deployment with SimpleSocketServer and Fluentbit - SimpleSocketServer is responsible for receiving logs from kafka, saving to files then fluetbit reads these files and sends to elastic.

Finally I have implemented second idea but have issue with determine from where logs come from. This is why I'm trying to use HOSTNAME.

[rkuccf]

@scholzj do you see any chance to add to Strimzi any way to determine the pod name in logs or possibility to define additional sidecar?

[scholzj]

I don't know. You can always open a proposal and see what people think, but I think there would need to be a good use-case that is generally applicable. I think in this case you should stick with the usual practices and collect the logs through the container engine.