395 Background processing [2d]

Some tasks in a web application are better not done live when a user requests a page, but in the background. Examples are longer running tasks, tasks that are not tied to user interaction, and tasks that can fail and may need to be retried. Our two main mechanisms for background processing are cronjobs (managed with whenever Show archive.org snapshot ) and job queues (ActiveJob, usually with GoodJob).

Important

Work on this lesson in builder mode.

Learning goals

• You can explain which work belongs in the background — long-running, not tied to a user interaction, or likely to fail and need a retry — and what changes in the UI when work becomes asynchronous.
• You can explain the two mechanisms we use for background work and when to use which: a job queue for work triggered by the application, cronjobs for work that runs on a schedule.
• You can explain what Active Job is: a Rails abstraction with a uniform interface over any queue implementation — and why you still need to know the underlying tool for anything non-standard.
• You can set up a queue in a Rails app that has none: install GoodJob, configure it as the Active Job adapter, and put its dashboard behind authentication.
• You can move work into a background job and explain how a job is stored, picked up, retried and reported when it fails.
• You can explain why a job must not be enqueued before the surrounding database transaction has committed, and how to do it right.
• You can schedule a cronjob with whenever, and explain what each run costs and on which servers it should run.
• You can test code that enqueues and performs jobs.
• You can compare GoodJob with the alternatives you will meet in other projects — Sidekiq (very mature, needs Redis) and Solid Queue (bundled with Rails, less mature) — and explain why GoodJob is our default.

Resources

Read what's new to you, skim what's familiar, skip what you already master. Stop when you can meet the learning goals.

Your agent can also generate an overview, a tutorial or an explanation for anything here, tailored to what you already know. Just ask.

Job queues with Active Job and GoodJob

• 📄 Rails Guide: Active Job Basics Show archive.org snapshot — the abstraction: jobs, queues, callbacks, retries, and transactional integrity Show archive.org snapshot
• 📄 GoodJob README Show archive.org snapshot — our queue: set up, configuration Show archive.org snapshot , the dashboard Show archive.org snapshot , how jobs are executed and retried
• 📄 Rails Guide: Testing jobs Show archive.org snapshot and the have_enqueued_job matcher Show archive.org snapshot — how to test enqueuing and performing
• 📄 Why jobs should never be enqueued in an after_create or after_save callback — our card (written for Sidekiq; the reasoning is the same for every queue)
• 📄 Using ActiveRecord with threads might use more database connections than you think — our card: what running jobs in threads means for your database

Cronjobs

• 📄 whenever Show archive.org snapshot — our tool: a Ruby DSL that writes the crontab on every deploy
• 📄 How to add jobs to cron under Linux Show archive.org snapshot — what whenever generates for you
• 📄 Decide whether cronjobs should run on one or all servers — our card

Alternatives

We use GoodJob Show archive.org snapshot , which stores jobs in PostgreSQL — no extra service to run. In other projects you will meet:

• 📄 Sidekiq Show archive.org snapshot — very mature, Redis-backed; common in older projects
• 📄 Solid Queue Show archive.org snapshot — Rails' default since 8.0, database-backed, younger
• 📄 Card Our Rails stack, and why it isn't Omakase

GoodJob also offers cron-style recurring jobs Show archive.org snapshot . We still schedule periodic work with whenever: cronjobs and a job queue solve different problems, and a crontab works even when no worker process is running.

Discuss with your mentor

• When moving a task to the background introduces new UI states — what does the user see while the work is pending, and when it failed?
• Workers run jobs in threads. What does that mean for database connections and for code that isn't thread-safe?
• Why must a job not be enqueued from an after_create or after_save callback? What happens if the transaction rolls back?
• Each cronjob run boots the Rails application and loads a CPU while doing so. What does that mean for how often a cronjob should run, and on which servers?
• How do we get notified when a job or a cronjob fails?
• How do background workers fit into our infrastructure — which process runs them, where, and what happens when a server fails?

Exercises

Set up GoodJob

MovieDB has no job queue yet.

• Add the good_job gem and configure it as the Active Job adapter (config.active_job.queue_adapter = :good_job).

• Run its migrations and start a worker process alongside your server.

  Tip

  Add the worker to Procfile.dev (e.g. worker: bundle exec good_job start) so bin/dev brings it up with the server and the asset build. Otherwise you have to remember a second terminal every time you touch jobs — and jobs that silently never run are a confusing thing to debug.

• Mount the GoodJob dashboard in your routes, restricted to admin users, and open it.

Move API calls to a background job

You've previously implemented calls to external APIs in your MovieDB.

• Choose one of those and move it into a background job. The automatic fetching of the release year is a good candidate.
• Make sure your code is (still) tested: one spec that the job gets enqueued, one that performing it does the right thing.
• Watch your job in the dashboard.
• Simulate the API being unreliable, e.g. with a random TimeoutError. Watch what happens to the job: retries, backoff, the error in the dashboard. Decide how many retries make sense and configure them.

Reject stale movies

• In State machines you created a review process for new movies.
• Write a cronjob with whenever that automatically rejects a movie if it hasn't been accepted for 7 days.
• Decide whether it should run on one or all servers, and make sure a failure would be noticed.