Add kubernetes metadata to fluentd/fluent bit sidecar

Background: When using fluentd or fluent-bit as a daemonset, we can make use of fluent-plugin-kubernetes_metadata_filter to retrieve rich metadata information and label them into our logs, such as pod name, namespace labels, pod labels… etc.

<filter kubernetes.var.log.containers.**.log>
  @type kubernetes_metadata
</filter>

But when running fluentd or fluent-bit as a sidecar container, since we don’t use a cluster-wide service account to run fluentd, pods aren’t authorized to ‘watch’ Kubernetes metadata information anymore.

Solution: Get Kubernetes metadata via downward API (you don’t have to install or set up any extra stuff to use this API), store the information in environment variables, then retrieve them in fluentd or fluent-bit configs.

Example yaml (partially)

• Add environment variables in your fluentd or fluent-bit container yaml

- name: fluentd-sidecar
  image: ..../fluentd...
  env:
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: POD_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: POD_LABELS
    valueFrom:
      fieldRef:
        fieldPath: metadata.labels['app']
  - name: NODE_NAME
    valueFrom:
      fieldRef:
      fieldPath: spec.nodeName

• Retrieve environment variables in your fluentd configmap

<filter kubernetes.** >
  @type record_transformer
  <record>
    NODE_NAME "#{ENV['NODE_NAME']}"
  </record>
  <record>
    POD_NAMESPACE "#{ENV['POD_NAMESPACE']}"
  </record>
  <record>
    POD_NAME "#{ENV['POD_NAME']}"
  </record>
</filter>

• Or, retrieve environment variables in your fluent-bit configmap

[FILTER]
Name modify
Match *
Add POD_NAME ${POD_NAME}
Add POD_NAMESPACE ${POD_NAMESPACE}
Add NODE_NAME ${NODE_NAME}

References:

• Downward API documentation: https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/