Carrierwave: How to migrate to another folder structure

A flat folder structure can be cool if you have only a few folders but can be painful for huge amounts. We recently had this issue in a project with more than 100.000 attachments, where we used a structure like this /attachments/123456789/file.pdf.

Even the ls command lasted several minutes to show us the content of the attachments folder.

So we decided to use a more hierarchical structure with a limited maximum of folder per layer. Here are a few tips how to migrate your files to their new location.

Old structure: /attachments/123456789/<some-secret>/file.pdf
New structure: /attachments/123/456/789/<some-secret>/file.pdf

Be sure to also read our card on migrating from Paperclip to Carrierwave, especially the list of possible errors that might be unexpected in the first place.

Step 1: Prepare your Uploader for the new structure

Implement a new structure, like described in a separate card.

We will later migrate existing files from the old directory structure to the new one. That will take some time and your application should be able to serve files from both storage locations in the meantime. To do that, make store_dir aware of it.

Copyclass FileUploader < CarrierWave::Uploader::Base
  def store_dir
    if Pathname.new(new_store_dir).directory? # already moved
      new_store_dir
    elsif Pathname.new(old_store_dir).directory? # not yet moved
      old_store_dir
    else # new uploads should use the new structure
      new_store_dir
    end
  end

  def new_store_dir
    # put your new logic here
  end

  def old_store_dir
    # move your previous store_dir code here
  end
end

If the files are already in their new folder, Carrierwave will look there. Otherwise it will use the old directory.

Step 2: Write a migration method to move your files

Now you have to move your file to their new directory. We have decided to move the secret folder with its contents.

Copydef migrate!
  correct_store_path = Pathname.new(new_store_dir)
  return if correct_store_path.directory? # already moved if the directory exists, nothing to do

  parent_path = correct_store_path.parent # this is the secret folder

  FileUtils.mkdir_p(parent_path) # create all necessary directories, e.g. /123/456/789
  FileUtils.mv(old_store_dir, new_store_dir) # move the directory to its new location

  old_store_path = Pathname.new(old_store_dir)
  begin
    FileUtils.rmdir(old_store_path.parent)  # cleanup empty directories afterwards
  rescue Errno::ENOTEMPTY => e # ignore exception for non-empty directories
    puts e.message
  end
end

Step 3: Deploy and execute migrate!

After deployment you can call your method in a script like this:

CopyAttachment.find_each { |attachment| attachment.file.migrate! }

with

Copyclass Attachment
 mount_uploader :file, FileUploader
end

Step 4: Remove temporary methods

Now you can replace the implementation of store_dir with new_store_dir and remove your temporary migrate! method.