A recent patch level Ruby update Show archive.org snapshot caused troubles to some of us as applications started to complain about incompatible gem versions. I'll try to explain how the faulty state most likely is achieved and how to fix it.
When you deploy a new Ruby version with capistrano-opscomplete Show archive.org snapshot , it will take care of a few things:
Now you should know that the gems for e.g. the Ruby version 2.6.5 are installed in this directory on our servers (relative to the current app directory)
../shared/bundle/ruby/2.6.0/gems
As you can see, the same gems are shared for all Ruby versions of the same minor level (e.g. 2.6.X). The patch level number is not included in the shared gem path.
If you update Ruby from 2.6.5 to 2.6.6, most gems will stay the same. A notable exception are gems with native extensions (like pg, nokogiri,..), which may be incompatible and must be re-built in this case. You will see a message such like this in the console during the deployment:
Ignoring nokogiri-1.10.4 because its extensions are not built. Try: gem pristine nokogiri --version 1.10.4
So whenever a patch level Ruby update is deployed, gems with native extensions are automatically re-installed. Because of the shared bundle path, only one patch-level Ruby version can have compatible gems at the same time.
Assuming we deploy a patch-level Ruby from 2.6.5 to 2.6.6, we end up in one of these situations:
Depending on the reason for the deployment failure, you can take different actions to bring the Ruby update to an end.
Bundler failed to install all gems
I observed that in some cases there seems to be a race condition when bundling all native extensions at once (with bundle install --jobs 4
).
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
./shared/bundle/ruby/2.6.0/gems/mini_racer-0.2.9/ext/mini_racer_extension
`initialize': No such file or directory @ rb_sysopen - ./shared/bundle/ruby/2.6.0/specifications/nokogumbo-1.5.0.gemspec (Errno::ENOENT)
In this case, deploying again should solve the issue.
A deploy task fails because of incompatible gem dependencies
This is most likely if your first deploy failed (but bundled the new versions), and the second deploy requires some of those gems. A common example is our "warn if there are any migrations" task. You will see an error like this:
01 bundle exec rake db:warn_if_pending_migrations
01 rake aborted!
01 LoadError: incompatible library version - ./shared/bundle/ruby/2.5.0/gems/pg-0.18.4/lib/pg_ext.so
01 ./shared/bundle/ruby/2.5.0/gems/pg-0.18.4/lib/pg.rb:4:in `require'
This is tricky. You are deploying from an old release directory (and thus an old Ruby version with incompatible gems), but can't deploy the new version. In this case, you can change the patch level of the .ruby-version
in the current
release directory of all servers of that deployment target to the new version and deploy again. Because on the first (failed) deploy, this version is the only supported one anyway. Don't do this unless you absolutely need to. There is a taks in capistrano-opscomplete-0.6.4
which can do that for you (see below).
capistrano-opscomplete-0.6.4
there were two tasks added to help with this problem. deploy:failed
:00:13 opscomplete:ruby:broken_gems_warning
WARN Deploy failed and the ruby version has been modified in this deploy.
WARN If this was a minor ruby version upgrade your running application may run into issues with native gem extensions.
WARN If your deploy failed before deploy:symlink:release you may run `bundle exec cap staging opscomplete:ruby:reset`.
WARN Please refer https://makandracards.com/makandra/477884-bundler-in-deploy-mode-shares-gems-between-patch-level-ruby-versions
.ruby-version
in current_path and runs bundle pristine
$ bundle exec cap staging opscomplete:ruby:reset
00:00 opscomplete:ruby:reset
01 rbenv global 2.7.1
✔ 01 deploy-capistrano-opscomplete_s@app01-stage.example.example.com 0.105s
02 bundle pristine
02 Installing rake 13.0.1
...
The task is only helpful if your deploy failed before the deploy:symlink:release
task. If it failed afterwards you need to roll back your release.