395 Background processing [2d]

Some tasks in a web application are better not done live when a user request a page, but in the background. Examples are

• longer running tasks
• tasks that are not tied to user interaction
• tasks that can fail, and may need to be retried

Our two main mechanisms for background processing are

• cronjobs (managed with whenever Show archive.org snapshot )
• job queues, usually with Sidekiq Show archive.org snapshot

Learn about cronjobs

• Read HowTo: Add Jobs To cron Under Linux or UNIX? Show archive.org snapshot
• Understand how we manage cronjobs with whenever Show archive.org snapshot .
• Find a project that uses whenever. Find out what kinds of work is done in those cronjobs. Can you find the code? The tests?
• What happens when an error occurs? How do we get notified?
• Understand that each call of a cron job will boot up your Rails application and produce 100% load on a CPU during booting. Talk to your mentor what that means for the load of your application.
• Decide whether cronjobs should run on one or all servers

Note

We use the whenever Show archive.org snapshot to automatically rewrite the crontab when we deploy. You have already intergrated whenever in your MovieDB in Consuming external APIs with Ruby.

Exercise

• In State machines you created a review process for new movies.
• Write a cronjob that automatically rejects a movie if it hasn't been accepted for 7 days.

Learn about Sidekiq

• Read Sidekiq's Readme Show archive.org snapshot
• Read Sidekiq best practices Show archive.org snapshot
• Read Sidekiq Error Handling Show archive.org snapshot
• Understand how to schedule jobs, and work them off
• Understand how ActiveJob integrates with Sidekiq
• What happens when an exception occurs? How do we get notified?
• Sidekiq uses threads
  • Discuss advantages and drawbacks with your mentor
  • Read Using ActiveRecord with threads might use more database connections than you think
• Sidekiq needs Redis
  • For what?
• Sidekiq jobs should never be enqueued in an after_create or after_save callback. Can you find out why? (Hint: You can find a spoiler here)
• Does it make sense to use both cronjobs and Sidekiq jobs in the same application?
• Find out how Sidekiq fits into our infrastructure. Is there failover? Where does it run?
• Talk to your mentor about: When moving a task to the background will introduce new UI states

Exercise: Move API calls to a Sidekiq job

You've previously implemented calls to external APIs in your MovieDB.

• Choose one of those and move it into a background job using Sidekiq. The automatic fetching of the release year is a good candidate.
• Make sure your code is (still) tested. Read the Sidekiq testing guide Show archive.org snapshot for some pointers.
• Sidekiq comes with a Web UI Show archive.org snapshot . Mount it into your MovieDB, so you can see your Sidekiq jobs under /sidekiq.
• Simulate the API being unreliable. Perhaps add a random TimeoutError. Can you see what happens to your job in the Web UI?
• The following cards will help you figure out how to start your Sidekiq during deployment:
  • How to upgrade to Sidekiq 6+
  • Setup Sidekiq and Redis
  • Using Sidekiq with Procfile and Capistrano (example)