Bash: How to grep logs for a pattern and expand it to the full request

Posted . Visible to the public.

Example

I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [53a240c1-489e-4936-bbeb-d6f77284cf38] nope
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [53a240c1-489e-4936-bbeb-d6f77284cf38] more

Goal

When searching through Rails logs on production, it's often hard to see all lines that belong to the same requests, since output of different requests is often interwoven. Instead, we want to find all requests that match a pattern, and then print all lines that share the request identifier.

Bash command

grep "test" test.log | awk '{ print $7}' | sed --regexp-extended 's/\[(.+)\]/\1/' | xargs | tr ' ' '|' | xargs --replace grep -E "{}" test.log

Result

I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more

Explain

  1. grep "test" test.log
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
  1. awk '{ print $7}'
[4cdad7a4-8617-4bc9-84e9-c40364eea2e4]
[6e047fb3-05df-4df7-808e-efa9fcd05f87]
  1. sed --regexp-extended 's/\[(.+)\]/\1/' [1]
4cdad7a4-8617-4bc9-84e9-c40364eea2e4
6e047fb3-05df-4df7-808e-efa9fcd05f87
  1. xargs
4cdad7a4-8617-4bc9-84e9-c40364eea2e4 6e047fb3-05df-4df7-808e-efa9fcd05f87
  1. tr ' ' '|'
4cdad7a4-8617-4bc9-84e9-c40364eea2e4|6e047fb3-05df-4df7-808e-efa9fcd05f87
  1. xargs --replace grep -E "{}" test.log [2]
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more
  • [1] -E, -r, --regexp-extended: use extended regular expressions in the script (for portability use POSIX -E).
  • [2] -i[replace-str], --replace[=replace-str]: This option is a synonym for -Ireplace-str if replace-str is specified. If the replace-str argument is missing, the effect is the same as -I{}. This option is deprecated; use -I instead.
Last edit
Tobias Kraze
License
Source code in this card is licensed under the MIT License.
Posted by Emanuel to makandra dev (2024-01-23 07:45)