Read more

How to search through logs on staging or production environments

Judith Roth
February 03, 2017Software engineer at makandra GmbH

We generally use multiple application servers (at least two) and you have to search on all of them if you don't know which one handled the request you are looking for.

Illustration book lover

Growing Rails Applications in Practice

Check out our e-book. Learn to structure large Ruby on Rails codebases with the tools you already know and love.

  • Introduce design conventions for controllers and user-facing models
  • Create a system for growth
  • Build applications to last
Read more Show archive.org snapshot

Rails application logs usually live in /var/www/<project-environment-name>/shared/log.
Web server logs usually live in /var/www/<project-environment-name>/log.

Searching through single logs with grep / zgrep

You can use grep in this directory to only search the latest logs or zgrep to also search older (already zipped) logs. zgrep is used just like grep and options will be passed to grep. For example

zgrep -B 2 "keyword" *

gives you two lines of context before every match for "keyword"

Running commands on multiple servers at the same time

With a custom Capistrano task:

CMD='zgrep -P "..." RAILS_ROOT/log/production.log' bundle exec cap production app:run_cmd

See Capistrano 3: Running a command on all servers.

Alternatives:

With Terminator

Use multiple shells in Terminator to ssh into each appliaction server. Then use the grouping feature of Terminator to send all keystrokes to all shells simultaneously. Show archive.org snapshot No need to chain the commands, just cd into the logs directory and then use grep / zgrep like in the example above.
This is usually our preferred way.

With custom rake task

You can add a rake task which tails logs from all remote servers. See this card.

With a service

If you need to do this often you may want to set up some services to collect your logs, like ELK (Elasticsearch, Logstash & Kibana), instead of using one of the solutions above.

Posted by Judith Roth to makandra dev (2017-02-03 15:51)