Fluent-bit — Using Kubernetes Enrichment Filter

Fluent-bit kubernetes Enrichment

Using Kubernetes brings the need to publish the PODs Logs.
Creating the logs brings the need to add the metadata of the specific POD.

This can be done via code but can be done easily via the Fluent-bit Kubernetes Filter.

Why I write this Post ?

When I used the Kubernetes Filter I got into some trouble and it took me a while to fix/understand.

Using Kubernetes Filter

Using the Filter can be a bit tricky.

Below is an example of Kubernetes Filter settings

    [FILTER]
        Name                kubernetes
        Match               kube.myservice.*
        Kube_Tag_Prefix     kube.myservice.var.log.containers
        Merge_Log           On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Buffer_Size         5MB

One of the important (and problematic as I find it) fields in the Kubernetes Filter is Kube_Tag_Prefix.

The default value for Kube_Tag_Prefix is kube.var.log.containers.

Kube_Tag_Prefix:
When the source records comes from Tail input plugin, this option allows to specify what’s the prefix used in Tail configuration.

What actually should be written and explained is that the Tag used by the INPUT section should be the prefix defined by the Kube_Tag_Prefix.

We can see the usage via the So following example of the FLuent-bit Pipeline:

Using an INPUT for a specific service myservice

• Process the INPUT with CRI PARSER
• Process the INPUTextracted message with JSON PARSER — getting the JSON log
• Process the JSON message via Kubernetes Filter to enrich the log data

Note,
It is important to ensure that the
hence

TAG=kube.myservice.* ⇒Kube_Tag_Prefix= kube.myservice.var.log.containers

[FILTER]
        Name                kubernetes
        Match               kube.myservice.*
        Kube_Tag_Prefix     kube.myservice.var.log.containers
        Merge_Log           On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Buffer_Size         5MB

The Final Fluent-bit Pipeline Config

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     debug
        Daemon        off
        Parsers_File  /fluent-bit/parsers/parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-stdout.conf
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.myservice.*
        Parser            cri
        Path              /var/log/containers/*-elastic-sync-*main*.log
        DB                /var/log/flb_kube_myservice.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Ignore_Older      1d
        Refresh_Interval  10
  filter-kubernetes.conf: |
    [FILTER]
        Name    parser
        Match   *
        key_name message
        Parser json

    [FILTER]
        Name                kubernetes
        Match               kube.myservice.*
        Kube_Tag_Prefix     kube.myservice.var.log.containers
        Merge_Log           On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Buffer_Size         5MB
   
  output-stdout.conf: |
    [OUTPUT]
        Name            stdout
        Match           kube.myservice.*

References

• Kubernetes Filter — https://docs.fluentbit.io/manual/pipeline/filters/kubernetes