kube-gelf

CoreOS kubernetes container logs & journald log collector with graylog output. Configurable through configmap and with provided cron example to mitigate some fluentd bugs as well as providing option for config reloads

kubectl create -f rbac.yaml

kubectl create configmap \
    --namespace kube-system kube-gelf \
    --from-file fluent.conf \
    --from-literal GELF_HOST=<server address> \
    --from-literal GELF_PORT=12201 \
    --from-literal GELF_PROTOCOL=<udp|tcp>

kubectl create -f daemonset.yaml

# optional, see notes below
kubectl create -f cron.yaml

After updating the configmap reloading fluentd config on all pods can be done with kubectl access. Please allow atleast a minute to pass before issuing the command due to Kubernetes not real-time syncing configmap updates to volumes.

for POD in `kubectl get pod --namespace kube-system -l app=kube-gelf | tail -n +2 | awk '{print $1}'`; do echo RELOAD ${POD}; kubectl exec --namespace kube-system ${POD} -- /bin/sh -c 'kill -1 1'; done

Cron

As of Kubernetes 1.8 batch/v1beta1 is enabled by default and no additional changes are needed.

If you are on < 1.8:

Enable batch/v2alpha1=true in the apiserver(s) --runtime-config= & restart apiservers + controller-manager. Also change the apiVersion from batch/v1beta1 to batch/v2alpha1

The cron.yaml can be used to deploy a cronJob that periodicly tells kube-gelf to reload it's configuration to also works around some fluend bugs.

I have several images made for different Kubernetes versions and you could adapt your cron.yaml by using any of my avail image tags here: https://hub.docker.com/r/roffe/kubectl/tags/

Fluentd Bugs

in_tail prevents docker from removing container fluent/fluentd#1680.

in_tail removes untracked file position during startup phase. It means the content of pos_file is growing until restart when you tails lots of files with dynamic path setting. I will fix this problem in the future. Check this issue. fluent/fluentd#1126.

Changelog

1.2

• Introduced new ENV variable GELF_PROTOCOL for protocol selection. Valid values are "udp" or "tcp". This requires a update of you configmap from earlier verisons
• Changed output plugin to https://github.com/bodhi-space/fluent-plugin-gelf-hs

1.1

Got rid of hostNetwork and added a NODENAME env variable utilizing the downward api. All log entries will contain the field hostname: <your nodename in kubernetes>