Posted almost 6 years ago. Visible to the public.
Background processing [2d]
Some tasks in a web application are better not done live when a user request a page, but in the background. Examples are
- longer running tasks
- tasks that are not tied to user interaction
- tasks that can fail, and may need to be retried
Our two main mechanisms for background processing are
- Read HowTo: Add Jobs To cron Under Linux or UNIX?
- Understand how we manage cronjobs with whenever.
- Find a project that uses whenever. Find out what kinds of work is done in those cronjobs. Can you find the code? The tests?
- What happens when an error occurs? How do we get notified?
- Understand that each call of a cron job will boot up your Rails application and produce 100% load on a CPU during booting. Talk to your mentor what that means for the load of your application.
- Decide whether cronjobs should run on one or all servers
- Read Sidekiq's Readme
- Read Sidekiq best practices
- Understand how to schedule jobs, and work them off
- Understand how ActiveJob integrates with Sidekiq
- What happens when an exception occurs? How do we get notified?
- Sidekiq uses threads
- Discuss advantages and drawbacks with your mentor
- How problematic do you think this is? To what do you have to pay attention?
- Sidekiq needs Redis
- For what?
- Why Redis (and not MySQL, or memcache)?
- Sidekiq jobs should never be enqueued in an
after_savecallback. Can you find out why?
- Can cronjobs and Sidekiq be combined? Name at least 2 reasons why this would make sense.
- Find out how Sidekiq fits into our infrastructure. Is there failover? Where does it run?
You've previously implemented calls to external APIs in your MovieDB.
- Choose one of those and move it into a background job using Sidekiq. The automatic fetching of the release year is a good candidate.
- Make sure your code is (still) tested. Read the Sidekiq testing guide for some pointers.
- Sidekiq comes with a Web UI. Mount it into your MovieDB, so you can see your Sidekiq jobs under
- Simulate the API being unreliable. Perhaps add a random
TimeoutError. Can you see what happens to your job in the Web UI?
- If you want to start your Sidekiq during deployment, use makandra-sidekiq.