Capistrano 3: Running a command on all servers

This Capistrano task runs a command on all servers.

bundle exec cap production app:run cmd='zgrep -P "..." RAILS_ROOT/log/production.log'

Code

# lib/capistrano/tasks/app.rake

namespace :app do

  # Use e.g. to grep logs on all servers:
  #   b cap production app:run_cmd cmd='zgrep -P "..." RAILS_ROOT/log/production.log' 
  #
  # * Use RAILS_ROOT as a placeholder for the remote Rails root directory.
  # * Append ` || test $? =1;` to grep calls in order to avoid exit code 1 (= "nothing found")
  # * To be able to process ...

Capistrano: creating a database dump if migrating

In Capistrano 3, your Capfile requires 'capistrano/rails/migrations', which brings two Capistrano tasks: deploy:migrate and deploy:migrating. The former checks whether migrations should be performed. If so, the latter is invoked, which performs the actual migrations.

Knowing this, it is easy to dump the db only if migrations will run. First, enable conditional migrations:

# config/deploy.rb
set :conditionally_migrate, true # Only attempt migration if db/migrate changed

Then hook up the dump task to deploy:migrating:

How to make sure that manual deploy tasks (scheduled in Pivotal Tracker) are executed on deploy (with Capistrano)

We regularly have tasks that need to be performed around a deploy. Be it to notify operations about changed application behavior, be it to run a little oneline script after the deploy. Most database-related stuff can be handled by migrations, but every once in a while, we have tasks that are much easier to be performed manually.

Writing deploy tasks

Here is how we manage the deploy tasks themselves:

  • Deploy tasks are written inside the Pivotal Tracker story description, clearly marked (e.g. with a headline "Deploy task")
  • We disting...

Local deployment after pipeline succeeds

If you have a fully functional CI pipeline but no CD, you might find yourself frequently waiting for CI (with "merge after pipeline succeeds") just to perform the deployment.

The following command waits for the next commit that lands on the current branch (should be main or similar) and proceeds to deploy staging afterwards:

alias await-deployment='watch -g git pull && bundle exec cap staging deploy'

Note

Use at your own risk.
You could be deploying code from someone else that was pushed to the same branch in the meantime.

Rails: Fixing ETags that never match

Every Rails response has a default ETag header. In theory this would enable caching for multiple requests to the same resource. Unfortunately the default ETags produced by Rails are effectively random, meaning they can never match a future request.

Understanding ETags

When your Rails app responds with ETag headers, future requests to the same URL can be answered with an empty response if the underlying content ha...

Heads up: Deployment with newly generated SSH key (using ED25519) might fail

If you use a newer SSH key generated with the ED25519 algorithm instead of RSA (see Create a new SSH key pair), the deployment with Capistrano may fail with the following message:

The deploy has failed with an error: unsupported key type `ssh-ed25519'
net-ssh requires the following gems for ed25519 support:
 * ed25519 (>= 1.2, < 2.0)
 * bcrypt_pbkdf (>= 1.0, < 2.0)
See https://github.com/net-ssh/net-ssh/issues/565 for more information
Gem::LoadError : "ed25519 i...

Story Checklist Template

This is an story checklist I use to work on stories. For this purpose I extracted several cards related to the makandra process and ported them into a check list and refined that over time a little bit.

This task list is divded by the Gate keeping process in the following steps:

1. Starting a new feature
2. Working on the story
3. Finishing a feature
4. After Review

Here are ...

Fix: esbuild assets are missing after capistrano deploy

Issue: You have an app using jsbundling-rails and esbuild. After deploy, the assets built by esbuild are missing in public/assets.

Solution: Add app/builds to your git repo (by adding a app/builds/.keep file).

Something in sprockets is caching paths and refuses to accept files in "unknown" locations.

net-ssh and openssl-3.0.0

You'll need openssl-3 or newer for servers running 22.04

Ruby version 3.1 uses by default the gem openssl-3.0.0. This can cause issues with the gem net-ssh (6.1.0). This is a known bug.

Typically this can cause an error while deploying an application with capistrano:

could not verify server signature (SSHKit::Runner::ExecuteError)

or

Ed25519::VerifyError: signature verification failed!

As temporary workaround add the following line to your Gemfile:

gem 'openssl', ...

Deployment: Merge consecutive commits without cherry-picking

You want to deploy new features but the latest commits are not ready for production? Then use git merge master~n to skip the n-last commits.

Tip

A big advantage of merging vs. cherry-picking is that cherry-picking will create copies of all picked commits. When you eventually do merge the branch after cherry-picking, you will have duplicate commit messages in your history.

Example

It's time for a production deployment!

git log --pretty=format:"%h - %s" --reverse origin/production..origin/master

0e6ab39f - Feature A
6396...

When does Webpacker compile?

Webpack builds can take a long time, so we only want to compile when needed.

This card shows what will cause Webpacker (the Rails/Webpack integration) to compile your assets.

When you run a dev server

While development it is recommended to boot a webpack dev server using bin/webpack-dev-server.

The dev server compiles once when booted. When you access your page on localhost before the initial compilation, the page may load without assets.

The ...

Capistrano: Deployment issue undefined method `[]' for nil:NilClass

In newer passenger versions the output of passenger -v has changed. capistrano-passenger tries to parse the version and now causes the error undefined method '[]' for nil:NilClass. To fix this you only need to upgrade the capistrano-passenger gem.

Therefore run bundle update capistrano-passenger --conservative.

The version change of passenger from 6.0.7 to 6.0.8 has triggered this problem. This is fixed in capistrano-passenger >= 0.2.1.

Delivering Carrierwave attachments to authorized users only

Preparation

To attach files to your records, you will need a new database column representing the filename of the file. To do this, add a new migration (rails g migration <name>) with the following content:

class AddAttachmentToNotes < ActiveRecord::Migration[6.0]
  def change
    add_column :notes, :attachment, :string
  end
end

Don't forget to rename the class and change the column details to fit your purpose. Run it.

1) Deliver attachments through Rails

The first way is to store your Carrierwave attachments not ...

How to fix "Command "webpack" not found"

I just ran into this deployment error after switching from the asset pipeline to webpack:

01:05 deploy:assets:precompile
      01 bundle exec rake assets:precompile
      01 Compiling...
      01 Compilation failed:
      01 yarn run v1.22.5
      01 error Command "webpack" not found.
rake stderr: Nothing written

The problem is not related to the "webpack" dependency. You probably just forgot to add a binstub to run "yarn install":

Add these lines to "bin/ya...

SSHKit 1.9.0 failure for Capistrano deploy

SSHKit 1.9.0 might fail with the following error, when trying to deploy a Rail application. Upgrading the gem to version 1.21.0 fixed the issue.

Traceback (most recent call last):
	17: from /home/user/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/sshkit-1.9.0/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
	16: from /home/user/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/sshkit-1.9.0/lib/sshkit/backends/abstract.rb:29:in `run'
	15: from /home/user/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/sshkit-1.9....

Whenever requires you to set the application attribute in the Capistrano config

Whenever requires you to set the application attribute in your Capistrano configuration. Otherwise your cronjobs are created multiple times.

Example entry in config/deploy.rb:

set :application, 'some-app' # allows "set :whenever_identifier, ->{ "#{fetch(:application)}_#{fetch(:stage)}" }" to work as expected

Good

Then the crontab -l output will look like this:

# Begin Whenever generated tasks for: som...

Configuring Webpacker deployments with Capistrano

When deploying a Rails application that is using Webpacker and Capistrano, there are a few configuration tweaks that optimize the experience.

Using capistrano-rails

capistrano-rails is a Gem that adds Rails specifics to Capistrano, i.e. support for Bundler, assets, and migrations. While it is designed for Asset Pipeline (Sprockets) assets, it can easily be configured for Webpacker. This brings these features to the Webpacker world:

  • Automatic removal of expired assets
  • Manifest backups

Carrierwave: How to migrate to another folder structure

A flat folder structure can be cool if you have only a few folders but can be painful for huge amounts. We recently had this issue in a project with more than 100.000 attachments, where we used a structure like this /attachments/123456789/file.pdf.

Even the ls command lasted several minutes to show us the content of the attachments folder.

So we decided to use a more hierarchical structure with a limited maximum of folder per layer. Here are a few tips how to migrate your files to their new...

Geordi 4 released

4.0.0 2020-07-30

Compatible changes

  • Improved documentation; README now includes command options.
  • Improvement #90: geordi console, geordi deploy, geordi rake and geordi shell now work correctly if the project hasn't been bundled before
  • Use binstubs if present – breaks Geordi execution when a binstub is not working

Breaking changes

  • Removed deprecated executables

whenever: Installing cron jobs only for a given Rails environment or Capistrano stage

We use the whenever gem to automatically update the crontab of the servers we deploy to. By default, whenever will update all servers with a matching role (we use the :cron role ).

This card describes how to install some tasks only for a given Rails environment or for a given Capistrano stage ("deployment target").

Installing jobs only for a given Rails environment
-----------------------------------...

Bundler in deploy mode shares gems between patch-level Ruby versions

A recent patch level Ruby update caused troubles to some of us as applications started to complain about incompatible gem versions. I'll try to explain how the faulty state most likely is achieved and how to fix it.

Theory

When you deploy a new Ruby version with capistrano-opscomplete, it will take care of a few things:

  • The new Ruby version is installed
  • The Bundler version stated in the Gemfil...

Fix for "Rails assets manifest file not found" in Capistrano deploy

If you use webpacker in your Rails application, and you have completely disabled Sprockets, you might get the following error when trying to deploy: Rails assets manifest file not found. This happens inside the deploy:assets:backup_manifest task.

This task comes from capistrano-rails. It is build for Sprockets and does not work with Webpacker out of the box.

Solution

Configure capistrano-rails to work with Webpacker

Alternative

If you are using capistrano-rails, but...

Capistrano: Finding out who deployed which revision of your application and when

Capistrano automatically logs each (successful) deployment into a file on your application servers.

It is located at the root of your server's project folder, i.e. the parent of releases and current, like so:

/var/www/your-project$ ls
current
log
releases
repo
revisions.log  <---  here
shared

Each line in that file contains the deployed branch, commit, release ID, and username (was read from the deploying user's machine):

$ tail -n3 revisions.log 
Branch master (at da45511bea63002ac2ff002d1692e09d0dd7cb88) deployed as rel...

Webpack(er): A primer

webpack is a very powerful asset bundler written in node.js to bundle (ES6) JavaScript modules, stylesheets, images, and other assets for consumption in browsers.

Webpacker is a wrapper around webpack that handles integration with Rails.

This is a short introduction.

Installation

If you haven't already, you need to install node.js and Yarn.

Then, put

gem 'webpacker', '~> 4.x' # check if 4.x is still cu...