Read more

Limiting CPU and memory resources of Paperclip convert jobs

Thomas Eisenbarth
December 23, 2011Software engineer at makandra GmbH

If you're using Paperclip to store and convert images attached to your models, processing a lot of images will probably cause headache for your system operation colleagues because CPU and/or memory peaking.

Illustration online protection

Rails Long Term Support

Rails LTS provides security patches for old versions of Ruby on Rails (2.3, 3.2, 4.2 and 5.2)

  • Prevents you from data breaches and liability risks
  • Upgrade at your own pace
  • Works with modern Rubies
Read more Show archive.org snapshot

If you're on Unix you can use nice to tell the Kernel scheduler to prefer other processes that request CPU cycles. Keep in mind that this will not help if you're running into memory or IO trouble because you saved some bucks when you ordered (slow) harddrives.

ImageMagick (the tool which is used by Paperclip to do all that funky image convertion stuff) has some build-in possibilities to address that problem.
ImageMagick is aware of some environment variables set, e.g. MAGICK_MAP_LIMIT to "maximum amount of memory map in bytes to allocate for the pixel cache" or MAGICK_MEMORY_LIMIT to set the maximum amount of memory in bytes to allocate for the pixel cache from the heap MAGICK_THROTTLE to "periodically yield the CPU for at least the time specified in milliseconds."

See http://www.imagemagick.org/script/resources.php Show archive.org snapshot for a full list.

If you're using Passenger and Apache, use SetEnv within the VHost in question:

SetEnv MAGICK_THROTTLE 100

Keep in mind that this will not affect processes that are invoked through the Rails console. Put the environment variables into the user's shell configuration (e.g., ~/.bashrc) if you want to limit this, too.

Posted by Thomas Eisenbarth to makandra dev (2011-12-23 12:25)