Posted 18 days ago. Visible to the public.

Carrierwave: How to migrate to another folder structure

A flat folder structure can be cool if you have only a few folders but can be painful for huge amounts. We recently had this issue in a project with more than 100.000 attachments, where we used a structure like this /attachments/123456789/file.pdf.

Even the ls command lasted several minutes to show us the content of the attachments folder.

So we decided to use a more hierarchical structure with a limited maximum of folder per layer. As our attachment folder will grow very fast we choosed to use three layers, but that's up to you. Here are a few tips how to migrate your files to their new location.

Old structure: /attachments/123456789/<some-secret>/file.pdf
New structure: /attachments/123/456/789/<some-secret>/file.pdf

Carrierwave uses a method called store_dir to calculate the path of uploads. For the old structure it looked like this:

Copy
def store_dir File.join [ base_path.join("#{Rails.env}#{ENV['TEST_ENV_NUMBER']}"), namespaced_model_name, model.id, secret_folder(model), # Prevent users from guessing file names ].map(&:to_s) end

For our desired structure we have to replace the model.id with the new hierarchy. Be sure to read this card first, especially the list of possible errors that might be unexpected in the first place.

Step 1: Prepare your Uploader for the new structure

The migration will last some time and your application should be able to handle new files and old files during this step. So you have to keep the old structure as fallback. The store_dir method is the right place for this:

Copy
class FileUploader < CarrierWave::Uploader::Base def store_dir if Pathname.new(new_dir).directory? # already moved new_dir elsif Pathname.new(old_dir).directory? # not yet moved old_dir else # new uploads should use the new structure new_dir end end end

If the files are already in their new folder, Carrierwave will look there otherwise it will use the old directory. The old_dir method looks the same as your old store_dir method. The new_dir uses *split_id_path(model) instead of model.id with

Copy
def split_id_path(model) padded_id = model.id.to_s.rjust(9, '0') padded_id.scan(/.../) end

Step 2: Write a migration method to move your files

Now you have to move your file to their new directory. We have decided to move the secret folder with its contents.

Copy
def migrate! correct_store_path = Pathname.new(new_dir) return if correct_store_path.directory? # already moved if the directory exists, nothing to do parent_path = correct_store_path.parent # this is the secret folder FileUtils.mkdir_p(parent_path) # create all necessary directories, e.g. /123/456/789 FileUtils.mv(old_dir, new_dir) # move the directory to its new location old_store_path = Pathname.new(old_dir) begin FileUtils.rmdir(old_store_path.parent) # cleanup empty directories afterwards rescue Errno::ENOTEMPTY => e # ignore exception for non-empty directories puts e.message end end

Step 3: Deploy and execute migrate!

After deployment you can call your method in a script like this:

Copy
Attachment.find_each { |attachment| attachment.file.migrate! }

with

Copy
class Attachment mount_uploader :file, FileUploader end

Step 4: Remove temporary methods

Now you can replace the implementation of store_dir with new_dir and remove your temporary migrate! method.

Growing Rails Applications in Practice
Check out our new e-book:
Learn to structure large Ruby on Rails codebases with the tools you already know and love.

Owner of this card:

Avatar
Florian Leinsinger
Last edit:
12 days ago
by Dominik Schöler
Keywords:
uploads, file, storage, migrating
About this deck:
We are makandra and do test-driven, agile Ruby on Rails software development.
License for source code
Posted by Florian Leinsinger to makandra dev
This website uses short-lived cookies to improve usability.
Accept or learn more