Posted 2 months ago. Visible to the public.

How to implement simple queue limiting/throttling for Sidekiq

The sidekiq-rate-limiter gem allows rate-limiting Sidekiq jobs and works like a charm. However, it needs to be integrated on a per-worker basis.

If you want to limit a whole queue instead, and if your requirements are simple enough, you can do it via a Sidekiq middleware yourself.

Here is an example that limits concurrency of the "mailers" queue to 1. It uses a database mutex via the with_advisory_lock gem but you could easily use something else.

Copy
class SidekiqMailersThrottle def call(worker, job_options, queue) if queue == 'mailers' ApplicationRecord.with_advisory_lock('mailers-throttle') do yield end else yield end end end Sidekiq.configure_server do |config| config.server_middleware do |chain| chain.add SidekiqMailersThrottle end end

Some hints:

  • Using this approach, workers will be blocked while jobs are waiting for the mutex to free up. The sidekiq-rate-limiter gem re-queues blocked jobs so that other jobs can be worked off in the meantime (which is smart). Depending on your use case, blocking workers might or might not be an issue.
  • To increase concurrency, you might do something as simple as naming your lock "mailers-throttle-#{rand(4)}" (not perfect, but might be good enough).
  • For rate-limiting (e.g. only 1 job per 5 seconds) you could name it "mailers-throttle-#{(Time.now.to_i / 5.0).to_i}".

Also note that Sidekiq allows configuring queue weights. This is a different approach and will not throttle a queue, but instead allow some queues to be picked up more often than others.
Finally, it might be an option to spawn a separate Sidekiq process to work off only your a specific queue, and limit its worker count.

Once an application no longer requires constant development, it needs periodic maintenance for stable and secure operation. makandra offers monthly maintenance contracts that let you focus on your business while we make sure the lights stay on.

Owner of this card:

Avatar
Arne Hartherz
Last edit:
2 months ago
by Arne Hartherz
About this deck:
We are makandra and do test-driven, agile Ruby on Rails software development.
License for source code
Posted by Arne Hartherz to makandra dev
This website uses short-lived cookies to improve usability.
Accept or learn more