Read more

Limiting CPU and memory resources of Paperclip convert jobs

Thomas Eisenbarth
December 23, 2011Software engineer at makandra GmbH

If you're using Paperclip to store and convert images attached to your models, processing a lot of images will probably cause headache for your system operation colleagues because CPU and/or memory peaking.

Illustration money motivation

Opscomplete powered by makandra brand

Save money by migrating from AWS to our fully managed hosting in Germany.

  • Trusted by over 100 customers
  • Ready to use with Ruby, Node.js, PHP
  • Proactive management by operations experts
Read more Show archive.org snapshot

If you're on Unix you can use nice to tell the Kernel scheduler to prefer other processes that request CPU cycles. Keep in mind that this will not help if you're running into memory or IO trouble because you saved some bucks when you ordered (slow) harddrives.

ImageMagick (the tool which is used by Paperclip to do all that funky image convertion stuff) has some build-in possibilities to address that problem.
ImageMagick is aware of some environment variables set, e.g. MAGICK_MAP_LIMIT to "maximum amount of memory map in bytes to allocate for the pixel cache" or MAGICK_MEMORY_LIMIT to set the maximum amount of memory in bytes to allocate for the pixel cache from the heap MAGICK_THROTTLE to "periodically yield the CPU for at least the time specified in milliseconds."

See http://www.imagemagick.org/script/resources.php Show archive.org snapshot for a full list.

If you're using Passenger and Apache, use SetEnv within the VHost in question:

SetEnv MAGICK_THROTTLE 100

Keep in mind that this will not affect processes that are invoked through the Rails console. Put the environment variables into the user's shell configuration (e.g., ~/.bashrc) if you want to limit this, too.

Posted by Thomas Eisenbarth to makandra dev (2011-12-23 12:25)