Carrierwave: How to migrate to another folder structure

A flat folder structure can be cool if you have only a few folders but can be painful for huge amounts. We recently had this issue in a project with more than 100.000 attachments, where we used a structure like this /attachments/123456789/file.pdf.

Even the ls command lasted several minutes to show us the content of the attachments folder.

So we decided to use a more hierarchical structure with a limited maximum of folder per layer. As our attachment folder will grow very fast we choosed to use three layers, but that's up to you. Here are a few tips how to migrate your files to their new location.

Old structure: /attachments/123456789/<some-secret>/file.pdf
New structure: /attachments/123/456/789/<some-secret>/file.pdf

Carrierwave uses a method called store_dir to calculate the path of uploads. For the old structure it looked like this:

def store_dir
  File.join [
    base_path.join("#{Rails.env}#{ENV['TEST_ENV_NUMBER']}"),
    namespaced_model_name,
    model.id,
    secret_folder(model), # Prevent users from guessing file names
  ].map(&:to_s)
end

For our desired structure we have to replace the model.id with the new hierarchy. Be sure to read this card first, especially the list of possible errors that might be unexpected in the first place.

Step 1: Prepare your Uploader for the new structure

The migration will last some time and your application should be able to handle new files and old files during this step. So you have to keep the old structure as fallback. The store_dir method is the right place for this:

class FileUploader < CarrierWave::Uploader::Base
  def store_dir
    if Pathname.new(new_dir).directory? # already moved
      new_dir
    elsif Pathname.new(old_dir).directory? # not yet moved
      old_dir
    else # new uploads should use the new structure
      new_dir
    end
  end
end

If the files are already in their new folder, Carrierwave will look there otherwise it will use the old directory. The old_dir method looks the same as your old store_dir method. The new_dir uses *split_id_path(model) instead of model.id with

def split_id_path(model)
  padded_id = model.id.to_s.rjust(9, '0')
  padded_id.scan(/.../)
end

Step 2: Write a migration method to move your files

Now you have to move your file to their new directory. We have decided to move the secret folder with its contents.

def migrate!
  correct_store_path = Pathname.new(new_dir)
  return if correct_store_path.directory? # already moved if the directory exists, nothing to do

  parent_path = correct_store_path.parent # this is the secret folder

  FileUtils.mkdir_p(parent_path) # create all necessary directories, e.g. /123/456/789
  FileUtils.mv(old_dir, new_dir) # move the directory to its new location

  old_store_path = Pathname.new(old_dir)
  begin
    FileUtils.rmdir(old_store_path.parent)  # cleanup empty directories afterwards
  rescue Errno::ENOTEMPTY => e # ignore exception for non-empty directories
    puts e.message
  end
end

Step 3: Deploy and execute migrate!

After deployment you can call your method in a script like this:

Attachment.find_each { |attachment| attachment.file.migrate! }

with

class Attachment
 mount_uploader :file, FileUploader
end

Step 4: Remove temporary methods

Now you can replace the implementation of store_dir with new_dir and remove your temporary migrate! method.

Florian Leinsinger about 1 month ago
This website uses short-lived cookies to improve usability.
Accept or learn more