Read more

Bash: How to grep logs for a pattern and expand it to the full request

Emanuel
January 23, 2024Software engineer at makandra GmbH

Example

I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [53a240c1-489e-4936-bbeb-d6f77284cf38] nope
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [53a240c1-489e-4936-bbeb-d6f77284cf38] more

Goal

When searching through Rails logs on production, it's often hard to see all lines that belong to the same requests, since output of different requests is often interwoven. Instead, we want to find all requests that match a pattern, and then print all lines that share the request identifier.

Bash command

grep "test" test.log | awk '{ print $7}' | sed --regexp-extended 's/\[(.+)\]/\1/' | xargs | tr ' ' '|' | xargs --replace grep -E "{}" test.log

Result

I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more

Explain

  1. grep "test" test.log
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
  1. awk '{ print $7}'
[4cdad7a4-8617-4bc9-84e9-c40364eea2e4]
[6e047fb3-05df-4df7-808e-efa9fcd05f87]
  1. sed --regexp-extended 's/\[(.+)\]/\1/' [1]
4cdad7a4-8617-4bc9-84e9-c40364eea2e4
6e047fb3-05df-4df7-808e-efa9fcd05f87
  1. xargs
4cdad7a4-8617-4bc9-84e9-c40364eea2e4 6e047fb3-05df-4df7-808e-efa9fcd05f87
  1. tr ' ' '|'
4cdad7a4-8617-4bc9-84e9-c40364eea2e4|6e047fb3-05df-4df7-808e-efa9fcd05f87
  1. xargs --replace grep -E "{}" test.log [2]
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more
  • [1] -E, -r, --regexp-extended: use extended regular expressions in the script (for portability use POSIX -E).
  • [2] -i[replace-str], --replace[=replace-str]: This option is a synonym for -Ireplace-str if replace-str is specified. If the replace-str argument is missing, the effect is the same as -I{}. This option is deprecated; use -I instead.
Illustration online protection

Rails Long Term Support

Rails LTS provides security patches for old versions of Ruby on Rails (2.3, 3.2, 4.2 and 5.2)

  • Prevents you from data breaches and liability risks
  • Upgrade at your own pace
  • Works with modern Rubies
Read more Show archive.org snapshot
Emanuel
January 23, 2024Software engineer at makandra GmbH
Posted by Emanuel to makandra dev (2024-01-23 08:45)