Posted 9 months ago. Visible to the public.

Carrierwave: How to migrate to another folder structure

A flat folder structure can be cool if you have only a few folders but can be painful for huge amounts. We recently had this issue in a project with more than 100.000 attachments, where we used a structure like this /attachments/123456789/file.pdf.

Even the ls command lasted several minutes to show us the content of the attachments folder.

So we decided to use a more hierarchical structure with a limited maximum of folder per layer. Here are a few tips how to migrate your files to their new location.

Old structure: /attachments/123456789/<some-secret>/file.pdf
New structure: /attachments/123/456/789/<some-secret>/file.pdf

Be sure to also read our card on migrating from Paperclip to Carrierwave, especially the list of possible errors that might be unexpected in the first place.

Step 1: Prepare your Uploader for the new structure

Implement a new structure, like described in a separate card.

We will later migrate existing files from the old directory structure to the new one. That will take some time and your application should be able to serve files from both storage locations in the meantime. To do that, make store_dir aware of it.

Copy
class FileUploader < CarrierWave::Uploader::Base def store_dir if Pathname.new(new_dir).directory? # already moved new_store_dir elsif Pathname.new(old_dir).directory? # not yet moved old_store_dir else # new uploads should use the new structure new_store_dir end end def new_store_dir # put your new logic here end def old_store_dir # move your previous store_dir code here end end

If the files are already in their new folder, Carrierwave will look there. Otherwise it will use the old directory.

Step 2: Write a migration method to move your files

Now you have to move your file to their new directory. We have decided to move the secret folder with its contents.

Copy
def migrate! correct_store_path = Pathname.new(new_dir) return if correct_store_path.directory? # already moved if the directory exists, nothing to do parent_path = correct_store_path.parent # this is the secret folder FileUtils.mkdir_p(parent_path) # create all necessary directories, e.g. /123/456/789 FileUtils.mv(old_dir, new_dir) # move the directory to its new location old_store_path = Pathname.new(old_dir) begin FileUtils.rmdir(old_store_path.parent) # cleanup empty directories afterwards rescue Errno::ENOTEMPTY => e # ignore exception for non-empty directories puts e.message end end

Step 3: Deploy and execute migrate!

After deployment you can call your method in a script like this:

Copy
Attachment.find_each { |attachment| attachment.file.migrate! }

with

Copy
class Attachment mount_uploader :file, FileUploader end

Step 4: Remove temporary methods

Now you can replace the implementation of store_dir with new_dir and remove your temporary migrate! method.

Once an application no longer requires constant development, it needs periodic maintenance for stable and secure operation. makandra offers monthly maintenance contracts that let you focus on your business while we make sure the lights stay on.

Owner of this card:

Avatar
Florian Leinsinger
Last edit:
about 1 month ago
by Arne Hartherz
Keywords:
uploads, file, storage, migrating
About this deck:
We are makandra and do test-driven, agile Ruby on Rails software development.
License for source code
Posted by Florian Leinsinger to makandra dev
This website uses short-lived cookies to improve usability.
Accept or learn more