Read more

Bash: How to grep logs for a pattern and expand it to the full request

Emanuel
January 23, 2024Software engineer at makandra GmbH

Example

I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [53a240c1-489e-4936-bbeb-d6f77284cf38] nope
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [53a240c1-489e-4936-bbeb-d6f77284cf38] more

Goal

When searching through Rails logs on production, it's often hard to see all lines that belong to the same requests, since output of different requests is often interwoven. Instead, we want to find all requests that match a pattern, and then print all lines that share the request identifier.

Bash command

grep "test" test.log | awk '{ print $7}' | sed --regexp-extended 's/\[(.+)\]/\1/' | xargs | tr ' ' '|' | xargs --replace grep -E "{}" test.log

Result

I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more

Explain

  1. grep "test" test.log
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
  1. awk '{ print $7}'
[4cdad7a4-8617-4bc9-84e9-c40364eea2e4]
[6e047fb3-05df-4df7-808e-efa9fcd05f87]
  1. sed --regexp-extended 's/\[(.+)\]/\1/' [1]
4cdad7a4-8617-4bc9-84e9-c40364eea2e4
6e047fb3-05df-4df7-808e-efa9fcd05f87
  1. xargs
4cdad7a4-8617-4bc9-84e9-c40364eea2e4 6e047fb3-05df-4df7-808e-efa9fcd05f87
  1. tr ' ' '|'
4cdad7a4-8617-4bc9-84e9-c40364eea2e4|6e047fb3-05df-4df7-808e-efa9fcd05f87
  1. xargs --replace grep -E "{}" test.log [2]
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more
  • [1] -E, -r, --regexp-extended: use extended regular expressions in the script (for portability use POSIX -E).
  • [2] -i[replace-str], --replace[=replace-str]: This option is a synonym for -Ireplace-str if replace-str is specified. If the replace-str argument is missing, the effect is the same as -I{}. This option is deprecated; use -I instead.
Illustration book lover

Growing Rails Applications in Practice

Check out our e-book. Learn to structure large Ruby on Rails codebases with the tools you already know and love.

  • Introduce design conventions for controllers and user-facing models
  • Create a system for growth
  • Build applications to last
Read more Show archive.org snapshot
Emanuel
January 23, 2024Software engineer at makandra GmbH
Posted by Emanuel to makandra dev (2024-01-23 08:45)