In order to have monitoring for Sidekiq (like queue sizes, last run of Sidekiq) your application should have a monitoring route which returns a json looking like this:
{
"sidekiq": {
"totals": {
"failed": 343938,
"processed": 117649167
},
"recent_history": {
"failed": {
"2016-11-06": 1,
"2016-11-07": 46,
"2016-11-08": 0,
"2016-11-09": 0,
"2016-11-10": 0
},
"processed": {
"2016-11-06": 230653,
"2016-11-07": 230701,
"2016-11-08": 230645,
"2016-11-09": 230645,
"2016-11-10": 163385
}
},
"queue_sizes": {
"dead": 0,
"retries": 0,
"monitoring": 0,
"low_priority": 0,
"mails": 0,
"default": 0,
"elasticsearch": 0,
"high_priority": 0,
"file_upload": 0,
"scheduled": 0
},
"active_workers": 0
},
"timestamps": {
"sidekiq_performed": "2016-11-10 17:51:03",
"whenever_ran": "2016-11-10 17:51:02",
"requested": "2016-11-10 17:56:45"
}
}
Our friends from bitcrowd built a gem for it https://github.com/bitcrowd/sidekiq_monitoring Show archive.org snapshot .
Manual steps
In your application you need - at least - the following:
A controller rendering the monitoring output as JSON
class MonitoringController < ApplicationController
def status
data = {
timestamps: {
whenever_ran: Redis.current.get('monitoring:timestamp:whenever_ran'),
sidekiq_performed: Redis.current.get('monitoring:timestamp:sidekiq_performed'),
requested: Time.now.to_s(:db),
},
sidekiq: {
active_workers: sidekiq_stats.workers_size,
queue_sizes: sidekiq_queue_sizes,
recent_history: {
processed: sidekiq_history.processed,
failed: sidekiq_history.failed
},
totals: {
processed: sidekiq_stats.processed,
failed: sidekiq_stats.failed
}
}
}
render json: data
end
private
def sidekiq_stats
@sidekiq_stats ||= Sidekiq::Stats.new
end
def sidekiq_history
@sidekiq_history ||= Sidekiq::Stats::History.new(5)
end
def sidekiq_queue_sizes
queue_sizes = sidekiq_stats.queues
queue_sizes.merge({
scheduled: sidekiq_stats.scheduled_size,
retries: sidekiq_stats.retry_size,
dead: sidekiq_stats.dead_size
})
end
end
A cronjob that puts a simple job to Sidekiq to ensure it is running
Put this into config/schedule.rb (if you're using whenever). Use whatever else to build the Cronjob on your server.
job_type :enqueue, 'cd :path && :environment_variable=:environment bundle exec bin/enqueue :task'
every 5.minutes do
enqueue 'monitoring'
end
The worker itself should live in app/workers/cron_workers/monitoring_worker.rb and look like that:
require_relative 'base_worker'
class CronWorkers::MonitoringWorker < CronWorkers::BaseWorker
def perform
Redis.current.set('monitoring:timestamp:sidekiq_performed', Time.now.to_s(:db))
end
end
And finally you need this in bin/enqueue:
#!/usr/bin/env ruby
require_relative '../config/initializers/sidekiq'
def enqueue(worker_class, args = [])
job = worker_class.sidekiq_options.merge('args' => args, 'class' => worker_class.to_s)
Sidekiq::Client.push(job)
end
case ARGV.shift
when 'monitoring'
require_relative '../config/initializers/redis'
require_relative '../app/workers/cron_workers/monitoring_worker'
Redis.current.set 'monitoring:timestamp:whenever_ran', Time.now.strftime('%Y-%m-%d %H:%M:%S')
enqueue(CronWorkers::MonitoringWorker)
end
Finally - and this is propably most important - you need a monitoring system (Icinga, Nagios, etc.) that checks the JSON output generated with the stuff above.
We built this into http://railscomplete.de Show archive.org snapshot .
