How to search through logs on staging or production environments

Updated . Posted . Visible to the public. Repeats.

We generally use multiple application servers (at least two) and you have to search on all of them if you don't know which one handled the request you are looking for.

Rails application logs usually live in /var/www/<project-environment-name>/shared/log.
Web server logs usually live in /var/www/<project-environment-name>/log.

Searching through single logs with grep / zgrep

You can use grep in this directory to only search the latest logs or zgrep to also search older (already zipped) logs. zgrep is used just like grep and options will be passed to grep. For example

zgrep -B 2 "keyword" *

gives you two lines of context before every match for "keyword"

Running commands on multiple servers at the same time

With a custom Capistrano task:

CMD='zgrep -P "..." RAILS_ROOT/log/production.log' bundle exec cap production app:run_cmd

See Capistrano 3: Running a command on all servers.

Alternatives:

With Terminator

Use multiple shells in Terminator to ssh into each appliaction server. Then use the grouping feature of Terminator to send all keystrokes to all shells simultaneously. Show archive.org snapshot No need to chain the commands, just cd into the logs directory and then use grep / zgrep like in the example above.
This is usually our preferred way.

With custom rake task

You can add a rake task which tails logs from all remote servers. See this card.

With a service

If you need to do this often you may want to set up some services to collect your logs, like ELK (Elasticsearch, Logstash & Kibana), instead of using one of the solutions above.

Last edit
Dominik Schöler
Keywords
view
License
Source code in this card is licensed under the MIT License.
Posted to makandra dev (2017-02-03 14:51)