Use find_in_batches to process many records without tearing down the server
Occasionally you need to do something directly on the server – like having all records recalculate something that cannot be done in a migration because it takes a long time.
Let’s say you do something like this:
Even though you may have been successful with this on your development machine or the staging server, keep in mind that production machines often hold a lot more records. Using
all may just work, even with lots of records, but when you iterate over such records and fetch associations for every object, those will be attached to them and kept in memory as well – which causes more and more memory to be consumed over time.
When you need to process many records prefer
Project.find_in_batches do |projects| projects.each do |project| project.recalculate_statistics! end end
If the default of 1000 records per batch is still too much just set your own size like
find_in_batches(:batch_size => 50).
When you only want to iterate in batches (and don’t need to do anything on the scope
find_in_batches gives you), you can use
find_each (it does the
each from above for you) as a shortcut:
Project.find_each do |project| project.recalculate_statistics! end
If you are on Rails 2.3, be aware that
find calls inside the block are implicitly scoped. This is fixed in Rails 3+.
Author of this card:
- Last edit:
6 months agoby Henning Koch
- About this deck:
- We are makandra and do test-driven, agile Ruby on Rails software development.
- License for source code
License for source code
All source code included in the card Use find_in_batches to process many records without tearing down the server is licensed under the license stated below. This includes both code snippets embedded in the card text and code that is included as a file attachment. Excepted from this license are code snippets that are explicitely marked as citations from another source.
The MIT License (MIT) Copyright (c) 2011-2014 makandra GmbH Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.