Bundler in deploy mode shares gems between patch-level Ruby versions

A recent patch level Ruby update Show archive.org snapshot caused troubles to some of us as applications started to complain about incompatible gem versions. I'll try to explain how the faulty state most likely is achieved and how to fix it.

Theory

When you deploy a new Ruby version with capistrano-opscomplete Show archive.org snapshot , it will take care of a few things:

• The new Ruby version is installed
• The Bundler version stated in the Gemfile.lock is installed
• Geordi is installed (for database dumps)
• The gems of the Gemfile.lock are installed for the new Ruby version

Now you should know that the gems for e.g. the Ruby version 2.6.5 are installed in this directory on our servers (relative to the current app directory)

• ../shared/bundle/ruby/2.6.0/gems

As you can see, the same gems are shared for all Ruby versions of the same minor level (e.g. 2.6.X). The patch level number is not included in the shared gem path.

If you update Ruby from 2.6.5 to 2.6.6, most gems will stay the same. A notable exception are gems with native extensions (like pg, nokogiri,..), which may be incompatible and must be re-built in this case. You will see a message such like this in the console during the deployment:

Copy
Ignoring nokogiri-1.10.4 because its extensions are not built. Try: gem pristine nokogiri --version 1.10.4

So whenever a patch level Ruby update is deployed, gems with native extensions are automatically re-installed. Because of the shared bundle path, only one patch-level Ruby version can have compatible gems at the same time.

Reality

Assuming we deploy a patch-level Ruby from 2.6.5 to 2.6.6, we end up in one of these situations:

• If the deployment was successful, Ruby 2.6.6 is installed with all required gems. The new release directory uses exactly this Ruby version and everything is fine.
• If the deployment fails for whatever reason, Capistrano will not touch the currently used release directory and roll back all changes. The old release director will be kept with Ruby 2.6.5, which is now incompatible with all gems that require native extensions

Depending on the reason for the deployment failure, you can take different actions to bring the Ruby update to an end.

Possible reasons for a failed deployment

• Bundler failed to install all gems

  I observed that in some cases there seems to be a race condition when bundling all native extensions at once (with bundle install --jobs 4).

  CopyGem::Ext::BuildError: ERROR: Failed to build gem native extension.
  ./shared/bundle/ruby/2.6.0/gems/mini_racer-0.2.9/ext/mini_racer_extension
  `initialize': No such file or directory @ rb_sysopen - ./shared/bundle/ruby/2.6.0/specifications/nokogumbo-1.5.0.gemspec (Errno::ENOENT)
  

In this case, deploying again should solve the issue.

• A deploy task fails because of incompatible gem dependencies

  This is most likely if your first deploy failed (but bundled the new versions), and the second deploy requires some of those gems. A common example is our "warn if there are any migrations" task. You will see an error like this:

  Copy    01 bundle exec rake db:warn_if_pending_migrations
      01 rake aborted!
      01 LoadError: incompatible library version - ./shared/bundle/ruby/2.5.0/gems/pg-0.18.4/lib/pg_ext.so
      01 ./shared/bundle/ruby/2.5.0/gems/pg-0.18.4/lib/pg.rb:4:in `require'
  

This is tricky. You are deploying from an old release directory (and thus an old Ruby version with incompatible gems), but can't deploy the new version. In this case, you can change the patch level of the .ruby-version in the current release directory of all servers of that deployment target to the new version and deploy again. Because on the first (failed) deploy, this version is the only supported one anyway. Don't do this unless you absolutely need to. There is a taks in capistrano-opscomplete-0.6.4 which can do that for you (see below).

In capistrano-opscomplete-0.6.4 there were two tasks added to help with this problem.

1. A warning tasks which will be executed after deploy:failed:

Copy00:13 opscomplete:ruby:broken_gems_warning
      WARN  Deploy failed and the ruby version has been modified in this deploy.
      WARN  If this was a minor ruby version upgrade your running application may run into issues with native gem extensions.
      WARN  If your deploy failed before deploy:symlink:release you may run `bundle exec cap staging opscomplete:ruby:reset`.
      WARN  Please refer https://makandracards.com/makandra/477884-bundler-in-deploy-mode-shares-gems-between-patch-level-ruby-versions

2. A reset task which sets the ruby version to the .ruby-version in current_path and runs bundle pristine

Copy$ bundle exec cap staging opscomplete:ruby:reset
00:00 opscomplete:ruby:reset
      01 rbenv global 2.7.1
    ✔ 01 deploy-capistrano-opscomplete_s@app01-stage.example.example.com 0.105s
      02 bundle pristine
      02 Installing rake 13.0.1
...

The task is only helpful if your deploy failed before the deploy:symlink:release task. If it failed afterwards you need to roll back your release.