Read more

Parsing multiline container logs with fluent-bit

Claus-Theodor Riegg
January 25, 2023Software engineer at makandra GmbH

If you have for e.g. a Java application which outputs multiline stack traces inside a container running in kubernetes you might wonder how you can concat alle these lines to a single log message in fluent-bit. If fluent-bit is receiving the log output directly you can just set the multiline.parser to java. But when you're reading the logs from /var/log/containers this is not possible, because every line of the log message is logged in the CRI log format. This means that every line of the log becomes a JSON object containing additional metadata.

Illustration online protection

Rails Long Term Support

Rails LTS provides security patches for old versions of Ruby on Rails (2.3, 3.2, 4.2 and 5.2)

  • Prevents you from data breaches and liability risks
  • Upgrade at your own pace
  • Works with modern Rubies
Read more Show archive.org snapshot

According to the fluent-bit docs Show archive.org snapshot there is currently no ideal solution for nested stack traces:

We are still working on extending support to do multiline for nested stack traces and such. Over the Fluent Bit v1.8.x release cycle we will be updating the documentation.

There is an interesting Github issue Show archive.org snapshot regarding this topic. The problem can be solved when using the Old multiline configuration parameters Show archive.org snapshot and Old Docker mode configuration parameters Show archive.org snapshot . For example:

  inputs: |
    [INPUT]
        Name               tail
        Tag                kube.*
        Path               /var/log/containers/*.log
        Read_from_head     true
        Docker_Mode        On
        Docker_Mode_Parser multi_line
        Parser             docker
        Mem_Buf_Limit      5MB
        
customParsers: |
      [PARSER]
        Name    multi_line
        Format  regex
        Regex   (?<log>^{"log":"\d{4}-\d{2}-\d{2}.*)
Posted by Claus-Theodor Riegg to makandra Operations (2023-01-25 12:24)