Bash: How to grep logs for a pattern and expand it to the full request

Example

I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [53a240c1-489e-4936-bbeb-d6f77284cf38] nope
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [53a240c1-489e-4936-bbeb-d6f77284cf38] more

Goal

When searching through Rails logs on production, it's often hard to see all lines that belong to the same requests, since output of different requests is often interwoven. Instead, we want to find all requests that match a pattern, and then print all lines that share the request identifier.

Bash command

grep "test" test.log | awk '{ print $7}' | sed --regexp-extended 's/\[(.+)\]/\1/' | xargs | tr ' ' '|' | xargs --replace grep -E "{}" test.log

Result

I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more

Explain

grep "test" test.log

I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test

awk '{ print $7}'

[4cdad7a4-8617-4bc9-84e9-c40364eea2e4]
[6e047fb3-05df-4df7-808e-efa9fcd05f87]

sed --regexp-extended 's/\[(.+)\]/\1/' [1]

4cdad7a4-8617-4bc9-84e9-c40364eea2e4
6e047fb3-05df-4df7-808e-efa9fcd05f87

xargs

4cdad7a4-8617-4bc9-84e9-c40364eea2e4 6e047fb3-05df-4df7-808e-efa9fcd05f87

tr ' ' '|'

4cdad7a4-8617-4bc9-84e9-c40364eea2e4|6e047fb3-05df-4df7-808e-efa9fcd05f87

xargs --replace grep -E "{}" test.log [2]

I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more

[1] -E, -r, --regexp-extended: use extended regular expressions in the script (for portability use POSIX -E).
[2] -i[replace-str], --replace[=replace-str]: This option is a synonym for -Ireplace-str if replace-str is specified. If the replace-str argument is missing, the effect is the same as -I{}. This option is deprecated; use -I instead.

Emanuel

makandra.com

Say thanks1

Last edit

2024-01-29

Tobias Kraze

License

Source code in this card is licensed under the MIT License.

Posted by Emanuel to makandra dev (2024-01-23 07:45)