Read more

How to search through logs on staging or production environments

Judith Roth
February 03, 2017Software engineer at makandra GmbH

We generally use multiple application servers (at least two) and you have to search on all of them if you don't know which one handled the request you are looking for.

Illustration online protection

Rails Long Term Support

Rails LTS provides security patches for old versions of Ruby on Rails (2.3, 3.2, 4.2 and 5.2)

  • Prevents you from data breaches and liability risks
  • Upgrade at your own pace
  • Works with modern Rubies
Read more Show archive.org snapshot

Rails application logs usually live in /var/www/<project-environment-name>/shared/log.
Web server logs usually live in /var/www/<project-environment-name>/log.

Searching through single logs with grep / zgrep

You can use grep in this directory to only search the latest logs or zgrep to also search older (already zipped) logs. zgrep is used just like grep and options will be passed to grep. For example

zgrep -B 2 "keyword" *

gives you two lines of context before every match for "keyword"

Running commands on multiple servers at the same time

With a custom Capistrano task:

CMD='zgrep -P "..." RAILS_ROOT/log/production.log' bundle exec cap production app:run_cmd

See Capistrano 3: Running a command on all servers.

Alternatives:

With Terminator

Use multiple shells in Terminator to ssh into each appliaction server. Then use the grouping feature of Terminator to send all keystrokes to all shells simultaneously. Show archive.org snapshot No need to chain the commands, just cd into the logs directory and then use grep / zgrep like in the example above.
This is usually our preferred way.

With custom rake task

You can add a rake task which tails logs from all remote servers. See this card.

With a service

If you need to do this often you may want to set up some services to collect your logs, like ELK (Elasticsearch, Logstash & Kibana), instead of using one of the solutions above.

Posted by Judith Roth to makandra dev (2017-02-03 15:51)