We generally use multiple application servers (at least two) and you have to search on all of them if you don't know which one handled the request you are looking for.
Rails application logs usually live in /var/www/<project-environment-name>/shared/log.
Web server logs usually live in /var/www/<project-environment-name>/log.
Searching through single logs with grep / zgrep
You can use grep in this directory to only search the latest logs or zgrep to also search older (already zipped) logs. zgrep is used just like grep and options will be passed to grep. For example
zgrep -B 2 "keyword" *
gives you two lines of context before every match for "keyword"
Running commands on multiple servers at the same time
With a custom Capistrano task:
CMD='zgrep -P "..." RAILS_ROOT/log/production.log' bundle exec cap production app:run_cmd
See Capistrano 3: Running a command on all servers.
Alternatives:
With Terminator
Use multiple shells in Terminator to ssh into each appliaction server.
Then use the grouping feature of Terminator to send all keystrokes to all shells simultaneously.
Show archive.org snapshot
No need to chain the commands, just cd into the logs directory and then use grep / zgrep like in the example above.
This is usually our preferred way.
With custom rake task
You can add a rake task which tails logs from all remote servers. See this card.
With a service
If you need to do this often you may want to set up some services to collect your logs, like ELK (Elasticsearch, Logstash & Kibana), instead of using one of the solutions above.
