Carrierwave: Using a nested directory structure for file system performance

When storing files for lots of records in the server's file system, Carrierwave's default store_dir approach may cause issues. Here is a simple solution that scales for a long while.

The default storage directory from the Carrierwave templates looks like so:

class ExampleUploader < CarrierWave::Uploader::Base
  def store_dir
    "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end
end

If you store files for 500k records, that store_dir's parent directory will have 500k sub-directories which will cause some serious headaches when trying to navigate the file system, e.g. via ls or rsync.

Solution

A simple, proven solution has been to split model.id into chunks. If you are using secrets in your directory structure, this is applicable as well.

class ExampleUploader < CarrierWave::Uploader::Base
  def store_dir
    Rails.root.join(
      'public/system', # or any other storage root that you may prefer
      split_id_path(model),
      secret_folder(model)
    ).to_s
  end

  def split_id_path(model)
    padded_id = model.id.to_s.rjust(6, '0')
    padded_id.split(/(\d\d\d)$/).join('/')
  end

  def secret_folder(model)
    # if you use secret folders, do your magic here
  end
end

Example structure

The resulting directory structure will be:

  • /app-root/public/system/users/avatar/000/001/... (1st record)
  • /app-root/public/system/users/avatar/000/002/... (2nd record)
  • ...
  • /app-root/public/system/users/avatar/000/999/... (999th record)
  • /app-root/public/system/users/avatar/001/000/... (1000th record)
  • ...
  • /app-root/public/system/users/avatar/999/999/... (999'999th record)
  • /app-root/public/system/users/avatar/1000/000/... (1 millionth record)

So if you have 500k records, you will still only have 500 directories inside /app-root/public/users/avatar/. And inside each of them, at most 1000 sub-directories.

But I have millions of files

If you expect to store a lot more records, simply introduce a third level (.../123/456/789/...).

  def split_id_path(model)
    padded_id = model.id.to_s.rjust(9, '0')
    padded_id.split(/(\d\d\d)$/).join('/')
  end

See also

Arne Hartherz about 1 month ago
This website uses short-lived cookies to improve usability.
Accept or learn more