Limiting CPU and memory resources of Paperclip convert jobs

Posted Over 12 years ago. Visible to the public.

If you're using Paperclip to store and convert images attached to your models, processing a lot of images will probably cause headache for your system operation colleagues because CPU and/or memory peaking.

If you're on Unix you can use nice to tell the Kernel scheduler to prefer other processes that request CPU cycles. Keep in mind that this will not help if you're running into memory or IO trouble because you saved some bucks when you ordered (slow) harddrives.

ImageMagick (the tool which is used by Paperclip to do all that funky image convertion stuff) has some build-in possibilities to address that problem.
ImageMagick is aware of some environment variables set, e.g. MAGICK_MAP_LIMIT to "maximum amount of memory map in bytes to allocate for the pixel cache" or MAGICK_MEMORY_LIMIT to set the maximum amount of memory in bytes to allocate for the pixel cache from the heap MAGICK_THROTTLE to "periodically yield the CPU for at least the time specified in milliseconds."

See http://www.imagemagick.org/script/resources.php Show archive.org snapshot for a full list.

If you're using Passenger and Apache, use SetEnv within the VHost in question:

SetEnv MAGICK_THROTTLE 100

Keep in mind that this will not affect processes that are invoked through the Rails console. Put the environment variables into the user's shell configuration (e.g., ~/.bashrc) if you want to limit this, too.

Thomas Eisenbarth
Last edit
About 12 years ago
License
Source code in this card is licensed under the MIT License.
Posted by Thomas Eisenbarth to makandra dev (2011-12-23 11:25)