Warning when fetching big result set from a database.

Posted Almost 8 years ago. Visible to the public.

With large data set we can run into memory issue. Here is an example.

>> Post.published.count
=> 25000

>> Post.where(published: true).each do |post|
     post.archive!
   end

# Loads 25000 posts in memory

Rails 5 adds warning when loading large data set

To mitigate issue shown above Rails 5 has added config.active_record.warn_on_records_fetched_greater_than. When this configuration is set to an integer value, any query that returns the number of records greater than the set limit, logs a warning.

config.active_record.warn_on_records_fetched_greater_than = 1500

>> Post.where(published: true).each do |post|
     post.archive!
   end

=> Query fetched 25000 Post records: SELECT "posts".* FROM "posts" WHERE "posts"."published" = ? [["published", true]]
   [#<Post id: 1, title: 'Rails', user_id: 1, created_at: "2016-02-11 11:32:32", updated_at: "2016-02-11 11:32:32", published: true>, #<Post id: 2, title: 'Ruby', user_id: 2, created_at: "2016-02-11 11:36:05", updated_at: "2016-02-11 11:36:05", published: true>,....]

This helps us find areas where potential problems exist and then we can replace inefficient queries with better ones.

config.active_record.warn_on_records_fetched_greater_than = 1500

>> Post.where(published: true).find_each do |post|
     post.archive!
   end

# No warning is logged
Alexander M
Last edit
Over 7 years ago
Alexander M
Tags
Posted by Alexander M to Ruby and RoR knowledge base (2016-05-02 15:50)