Read more

Parsing multiline container logs with fluent-bit

Claus-Theodor Riegg
January 25, 2023Software engineer at makandra GmbH

If you have for e.g. a Java application which outputs multiline stack traces inside a container running in kubernetes you might wonder how you can concat alle these lines to a single log message in fluent-bit. If fluent-bit is receiving the log output directly you can just set the multiline.parser to java. But when you're reading the logs from /var/log/containers this is not possible, because every line of the log message is logged in the CRI log format. This means that every line of the log becomes a JSON object containing additional metadata.

Illustration web development

Do you need DevOps-experts?

Your development team has a full backlog? No time for infrastructure architecture? Our DevOps team is ready to support you!

  • We build reliable cloud solutions with Infrastructure as code
  • We are experts in security, Linux and databases
  • We support your dev team to perform
Read more Show snapshot

According to the fluent-bit docs Show snapshot there is currently no ideal solution for nested stack traces:

We are still working on extending support to do multiline for nested stack traces and such. Over the Fluent Bit v1.8.x release cycle we will be updating the documentation.

There is an interesting Github issue Show snapshot regarding this topic. The problem can be solved when using the Old multiline configuration parameters Show snapshot and Old Docker mode configuration parameters Show snapshot . For example:

  inputs: |
        Name               tail
        Tag                kube.*
        Path               /var/log/containers/*.log
        Read_from_head     true
        Docker_Mode        On
        Docker_Mode_Parser multi_line
        Parser             docker
        Mem_Buf_Limit      5MB
customParsers: |
        Name    multi_line
        Format  regex
        Regex   (?<log>^{"log":"\d{4}-\d{2}-\d{2}.*)
Posted by Claus-Theodor Riegg to makandra Operations (2023-01-25 12:24)