Parsing multiline container logs with fluent-bit

If you have for e.g. a Java application which outputs multiline stack traces inside a container running in kubernetes you might wonder how you can concat alle these lines to a single log message in fluent-bit. If fluent-bit is receiving the log output directly you can just set the multiline.parser to java. But when you're reading the logs from /var/log/containers this is not possible, because every line of the log message is logged in the CRI log format. This means that every line of the log becomes a JSON object containing additional metadata.

According to the fluent-bit docs Show archive.org snapshot there is currently no ideal solution for nested stack traces:

We are still working on extending support to do multiline for nested stack traces and such. Over the Fluent Bit v1.8.x release cycle we will be updating the documentation.

There is an interesting Github issue Show archive.org snapshot regarding this topic. The problem can be solved when using the Old multiline configuration parameters Show archive.org snapshot and Old Docker mode configuration parameters Show archive.org snapshot . For example:

Copy  inputs: |
    [INPUT]
        Name               tail
        Tag                kube.*
        Path               /var/log/containers/*.log
        Read_from_head     true
        Docker_Mode        On
        Docker_Mode_Parser multi_line
        Parser             docker
        Mem_Buf_Limit      5MB
        
customParsers: |
      [PARSER]
        Name    multi_line
        Format  regex
        Regex   (?<log>^{"log":"\d{4}-\d{2}-\d{2}.*)