HaProxy log format doesn't work for TCP frontend

[aimbot31]

Hello,

I'm trying to configure HaProxy to send logs in a JSON format with a format like this for example :
log-format: "{\"client_ip\":\"%ci\",\"client_port\":\"%cp\",\"frontend_port\":\"%fp\",\"date_time\":\"%t\",\"backend_name\":\"%b\",\"server_name\":\"%s\",\"Tt\":\"%Tt\",\"Tc\":\"%Tc\",\"Tw\":\"%Tw\",\"bytes_read\":\"%B\"}"

We cannot have that kind of format because of this line

kubernetes-ingress/pkg/handler/tcp-services.go

Line 124 in d404fc5

func (handler TCPServices) createTCPFrontend(h haproxy.HAProxy, frontendName, bindPort string, sslOffload bool) (frontend models.Frontend, reload bool, err error) {

which is adding option tcplog by
default even if we define logformat in global/defaults.

Am I missing something ? If this is not possible for the moment, how can we add this improvement to the ingress controller ?

[ivanmatmati]

Hi @aimbot31 , I'll look at the code. Keep you updated.

[aimbot31]

Hello @ivanmatmati thanks for the answer, did you find something ?

[ivanmatmati]

Hi @aimbot31 , we're going to implement a second log format annotation on tcp configmap so that you can have a specific one for TCP services (because of some incompatibilities with HTTP format).

[aimbot31]

@ivanmatmati this is great ! this feature will really help us ! Looking forward to see it

[fabianonunes]

Hello, @ivanmatmati.

Instead of utilizing annotations in the TCP ConfigMap, I've submitted the PR #570 that uses the main ConfigMap, allowing us to centralize the configurations in a single location.

[ivanmatmati]

Hi @fabianonunes , Thanks for being so active, it's very cool. Concerning this PR, I've already done the job. I think it'd be better if you see that something is scheduled that you ask to volunteer so that I can let you go on if ok. In this PR, there's three issues. First, it's better to keep all the TCP configuration in a single dedicated place (think SRP principle). Second, the format for log seems to require escaped spaces or additional quotes which does not align with existing log-format. Third, when a tcp-log-format is removed you must reinsert the tcplog option otherwise it will collide with log-format option (some of them are purely HTTP and so incompatible with TCP). Here I get an error when I remove the annotation. Thanks again, I still need to review your others PRs.

[ivanmatmati]

Hi @aimbot31 , the feature is in the master branch. You can test it with the nightly build. Just add/replace the version of your controller docker image with nightly. You will the documentation here.

[aimbot31]

Hello @ivanmatmati, will test it right away, again thanks for the reactivity, this was a big hole for us ! 💪

[aimbot31]

Hey again @ivanmatmati Seems not working on my server, what i did :

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-haproxy-kubernetes-ingress
  namespace: ingress-haproxy
  annotations:
    haproxy.com/log-format-tcp: "{\"client_ip\":\"%ci\",\"client_port\":\"%cp\",\"frontend_port\":\"%fp\",\"date_time\":\"%t\",\"backend_name\":\"%b\",\"server_name\":\"%s\",\"Tt\":\"%Tt\",\"Tc\":\"%Tc\",\"Tw\":\"%Tw\",\"bytes_read\":\"%B\"}"
data:
  log-format-tcp: "{\"client_ip\":\"%ci\",\"client_port\":\"%cp\",\"frontend_port\":\"%fp\",\"date_time\":\"%t\",\"backend_name\":\"%b\",\"server_name\":\"%s\",\"Tt\":\"%Tt\",\"Tc\":\"%Tc\",\"Tw\":\"%Tw\",\"bytes_read\":\"%B\"}"
  syslog-server: "address: fluentd-forwarder-syslog-svc.fluentd-forwarder.svc.cluster.local, port:5140, format: raw, facility:local0, length:4096"

And deployed like that

values: |
           controller:
              image:
                tag: nightly
              extraArgs:
                - --configmap-tcp-services=ingress-haproxy/public-smtp
                - --log=debug
                - --disable-ipv6

With the configmap for tcp services looking like that :

apiVersion: v1
kind: ConfigMap
metadata:
  name: public-smtp
  namespace: ingress-haproxy
  annotations:
    haproxy.com/log-format-tcp: "{\"client_ip\":\"%ci\",\"client_port\":\"%cp\",\"frontend_port\":\"%fp\",\"date_time\":\"%t\",\"backend_name\":\"%b\",\"server_name\":\"%s\",\"Tt\":\"%Tt\",\"Tc\":\"%Tc\",\"Tw\":\"%Tw\",\"bytes_read\":\"%B\"}"
    argocd.argoproj.io/sync-wave: "-1"
data:
  587:
    public-smtp-server/public-smtp-server:587
  465:
    public-smtp-server/public-smtp-server:465:ssl
  2525:
    public-smtp-server/public-smtp-server:2525

What i can see in the configuration when i jump inside the pod :

frontend tcp-2525
  mode tcp
  bind 0.0.0.0:2525 name v4
  option tcplog
  default_backend public-smtp-server_public-smtp-server_smtp2525

frontend tcp-465
  mode tcp
  bind 0.0.0.0:465 name v4 crt /etc/haproxy/certs/frontend ssl
  option tcplog
  default_backend public-smtp-server_public-smtp-server_smtp1465

frontend tcp-587
  mode tcp
  bind 0.0.0.0:587 name v4
  option tcplog
  default_backend public-smtp-server_public-smtp-server_smtp1587

haproxy ingress controller version :

/ $ ./haproxy-ingress-controller --version
2023/09/12 15:33:46 maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
HAProxy Ingress Controller v1.10.0 4d9884b.dev.dirtyBuild from: https://github.com/haproxytech/kubernetes-ingressGit commit date: 2023/09/02 16:16:38/ $ 

Do not hesitate if you require more informations

[aimbot31]

Hello @ivanmatmati If you have some time to tell me if i got something wrong because i haven't been able to make it work

[ivanmatmati]

Hi, as the documentation says the annotation is available only in TCP configmap not in the general one. Be aware that we kept the word annotation but actually it's data in the configmap. For instance, this would work:

apiVersion: v1
data:
  "6500": haproxy-controller/redis:6379
  log-format-tcp: '{"client_ip":"%ci","client_port":"%cp","frontend_port":"%fp","date_time":"%t","backend_name":"%b","server_name":"%s","Tt":"%Tt","Tc":"%Tc","Tw":"%Tw","bytes_read":"%B"}'
kind: ConfigMap
metadata:
  name: haproxy-kubernetes-ingress-tcp
  namespace: haproxy-controller