Use a label to determine whether to forward logs or not with Fluentd daemonset in K8s

While Implementing logging architecture in Kubernetes, we often run Fluentd as a daemonset to collect console logs from all pods and ship logs into EFK (or other storage). Usually, Fluentd daemonset is a cluster-level deployment, in which the config is shared by all namespaces and pods.

Under some circumstances, we’ll like to give the decision back to pods, so pod owners can decide whether they want their console logs to be shipped and stored.

To achieve this goal, pod owners should add a label to a pod, for example, we named it console_log_forward, with a value noforward, any other value strings will be ignored.

template:
    metadata:
      labels:
        app: myapp
        console_log_forward: noforward

And in Fluentd daemonset’s fluentd.config, add a new match paragraph with @type rewrite_tag_filter.

Explanation of this config is: When matching pattern (/^noforward$/) in the field ($.kubernetes.labels.console_log_forward), rewrite the log tag into “clear”.

And we need to add another match paragraph, when matching tag “clear”, use @type null to drop it.

<match kubernetes.**>
        @type rewrite_tag_filter
        <rule>
          key $.kubernetes.labels.console_log_forward
          pattern /^noforward$/
          tag clear
        </rule>
       # If there's other rewrite rules, add them here
</match>
<match clear>
    @type null
</match>

In this example case,

• Pods with console_log_forward: noforward label => pod logs will NOT be forward by fluentd.
• Pods without console_log_forward label => pod logs will not enter the above <match>, logs with continually be handled by the rest of the fluentd.config
• Pods with console_log_forward label but with a value other than “noforward” => pod logs will not enter the above <match>, log with continually be handled by the rest of your fluentd.config.