Best practices: Writing a Rails script (and how to test it)

A Rails script lives in lib/scripts and is run with bin/rails runner lib/scripts/.... They are a simple tool to perform some one-time actions on your Rails application. A Rails script has a few advantages over pasting some prepared code into a Rails console:

  • Version control
  • Part of the repository, so you can build on previous scripts for a similar task
  • You can have tests (see below)

Although not part of the application, your script is code and should adhere to the common quality standards (e.g. no spaghetti code). However, a script...

Bash: How to count and sort requests by IP from the access logs

Example

87.140.79.42 - - [23/Jan/2024:09:00:46 +0100] "GET /monitoring/pings/ HTTP/1.1" 200 814 "-" "Ruby"
87.140.79.42 - - [23/Jan/2024:09:00:46 +0100] "GET /monitoring/pings/ HTTP/1.1" 200 814 "-" "Ruby"
87.140.79.41 - - [23/Jan/2024:09:00:46 +0100] "GET /monitoring/pings/ HTTP/1.1" 200 814 "-" "Ruby"
87.140.79.42 - - [23/Jan/2024:09:00:46 +0100] "GET /monitoring/pings/ HTTP/1.1" 200 814 "-" "Ruby"

Goal

Count and sort the number of requests for a single IP address.

Bash Command

awk '{ print $1}' test.log | sort...

Bash: How to grep logs for a pattern and expand it to the full request

Example

I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [53a240c1-489e-4936-bbeb-d6f77284cf38] nope
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- ...

open-next-failure: An alias to speed up test debugging

Getting an entire test suite green can be a tedious task which involves frequent switches between the CLI that is running tests back to the IDE where its cause can be fixed.

The following bash aliases helped me speed up that process:

alias show-next-failure="bundle exec rspec --next-failure"
alias open-next-failure="show-next-failure || show-next-failure --format json  | jq -r '.examples[0]' | jq '\"--line \" + (.line_number|tostring) + \" \" + .file_path' | xargs echo | xargs rubymine"

There is a lot going on above but the gist...

Capistrano 3: Running a command on all servers

This Capistrano task runs a command on all servers.

bundle exec cap production app:run cmd='zgrep -P "..." RAILS_ROOT/log/production.log'

Code

# lib/capistrano/tasks/app.rake

namespace :app do

  # Use e.g. to grep logs on all servers:
  #   b cap production app:run_cmd cmd='zgrep -P "..." RAILS_ROOT/log/production.log' 
  #
  # * Use RAILS_ROOT as a placeholder for the remote Rails root directory.
  # * Append ` || test $? =1;` to grep calls in order to avoid exit code 1 (= "nothing found")
  # * To be able to process ...

Debug MiniMagick calls in your Rails app

Most of our applications use CarrierWave for file uploads. CarrierWave has an integrated processing mechanism for different file versions with support for ImageMagick through CarrierWave::MiniMagick (which requires the mini_magick gem). In case your processing runs into an error, CarrierWave will just swallow it and rethrow an error with a very generic message like Processing failed. Maybe it is not an image? which does not help you finding out what the actual problem is. CarrierWave probably does this for security purposes, but does n...

How to make sure that manual deploy tasks (scheduled in Pivotal Tracker) are executed on deploy (with Capistrano)

We regularly have tasks that need to be performed around a deploy. Be it to notify operations about changed application behavior, be it to run a little oneline script after the deploy. Most database-related stuff can be handled by migrations, but every once in a while, we have tasks that are much easier to be performed manually.

Writing deploy tasks

Here is how we manage the deploy tasks themselves:

  • Deploy tasks are written inside the Pivotal Tracker story description, clearly marked (e.g. with a headline "Deploy task")
  • We disting...

CarrierWave: Processing images with libvips

When you write your next CarrierWave uploader, consider processing your images with libvips instead of ImageMagick.

Reasons for libvips

There are several upsides to using libvips over ImageMagick:

How to pretty print all values in a Redis database

With this Ruby script you can print all values in a Redis database to your console (derived from this bash script).

Note: Do not run this command in production. This is for debugging purposes only.

def pretty_print_redis(redis)
  redis.keys.each_with_object({}) do |key, hash|
    type = redis.type(key)

    hash[key] = case type
    when 'string'
      redis.get(key)
    when 'hash'
      redis.hgetall(key)
    when 'list'
      redis.lrange(key, 0, -1)
    when 'set'
      redis.smembers(...

Sidekiq 7: Rate limiting with capsules

Sidekiq 7 adds a new feature called capsules.

Use cases:

  • a chrome queue limited to 1 for e.g. PDF processing to not overload the application server
  • an api queue, that limits a queue to 2 to protect the API server from too many requests in parallel

Example:

Sidekiq.configure_server do |config|
  # Edits the default capsule
  config.queues = %w[critical default low]
  config.concurrency = 5

  # Define a new capsule which ...

Sass: get rid of deprecation warnings in dependencies

TLDR: sass >= 1.35.0 has the option quietDeps to silence deprecation warnings from dependencies.

quietDeps: If true, the compiler must not print deprecation warnings
for stylesheets that are transitively loaded through an import path or importer.


You might have seen deprecation warnings like this during assets compilation:

DEPRECATION WARNING: Using / for division is deprecated and will be removed in Dart Sass 2.0.0.
Recommendation: math.div($grid-gutter-width, 2)
More info and automated migrator: https://sass-la...

Git: Restore

tl;dr

git checkout is the swiss army of git commands. If you prefer a semantically more meaningful command for restoring tasks, use git restore instead.

With this command you can ...

  • ... do unstaging - git restore --staged
  • ... discard staged changes - git restore --staged --worktree
  • ... discard unstaged changes - git restore
  • ... restore deleted files - git restore
  • ... restore historic versions - git restore --source
  • ... recreate merge conflicts - git restore --merge
  • ... specifiy...

How to fix "Exit with code 1 due to network error: ProtocolUnknownError" with wkhtmltopdf

New versions of wkhtmltopdf dissallow file:// URLs by default. You can allow them by passing --enable-local-file-access.

If you are using PDFKit, set the option

PDFKit.configure do |config|
  config.default_options = {
    enable_local_file_access: true,
  }
end

This will be necessary in many setups to allow wkhtmltopdf to fetch assets (such as stylesheets) from the filesystem.

Note on security

Allowing this poses some risk when you render user input, since it might be feasible to include data from the local filesyste...

Converting SVG to other vector formats without Inkscape

If you need to convert an SVG source to PS or EPS, the most common suggestion on the interwebs is to use Inkscape from the commandline.
Inkscape is a fairly resource-heavy tool with lots of dependencies. A great alternative for converting is CairoSVG.

CairoSVG is available on most Linux distros through their package management systems, e.g. apt install cairosvg on Ubuntu.
It has few dependencies (most importantly Python 3 and some related packages, but really not much)...

How to debug issues with zeitwerk and Rails

In case you have trouble with the zeitwerk autoloader, you can check out the documentation Autoloading and Reloading Constants and Classic to Zeitwerk HOWTO for some debugging hints.

For myself it was useful to print the registered constants and the file references during the boot. Therefore you need to add Rails.autoloaders.log! at the end of your config/application.rb file. You could also run `bin...

How to get information about a gem (via CLI or at runtime from Ruby)

When you need information about a gem (like version(s) or install path(s)), you can use the gem binary from the command line, or the Gem API inside a ruby process at runtime.

gem binary (in a terminal)

You can get some information about a gem by running gem info <gem name> in your terminal.

Example:

$ gem info irb

*** LOCAL GEMS ***

irb (1.4.1, 1.3.5)
    Author: Keiju ISHITSUKA
    Homepage: https://github.com/ruby/irb
    Licenses: Ruby, BSD-2-Clause
    Installed at (1.4.1): /home/arne/.rbenv/versions/3.0.3/lib/ruby/g...

Generating an Entity Relationship Diagram for your Rails application

This card explains how to generate an entity relationship diagram for your Rails application.
We also show how to limit your ERD to a subset of models, e.g. models inside a namespace.

Generating a full ERD

Option A: RubyMine

  1. Right-click anywhere in your project tree
  2. In the context menu, find the "Diagrams" menu item at/near the bottom
  3. Inside, choose "Show diagram" → "Rails Model Dependency Diagram"
  4. A new tab will open with the diagram inside. You can modify it there, and export it as an image.

Option B: Use rails-e...

Using feature flags to stabilize flaky E2E tests

A flaky test is a test that is often green, but sometimes red. It may only fail on some PCs, or only when the entire test suite is run.

There are many causes for flaky tests. This card focuses on a specific class of feature with heavy side effects, mostly on on the UI. Features like the following can amplify your flakiness issues by unexpectedly changing elements, causing excessive requests or other timing issues:

  • Lazy loading images
  • Autocomplete in search f...

Git shortcut to rebase onto another branch

Inspired by recent "git shortcut" cards I figured it would be nice to have one of these for rebasing a few commits onto another branch. The usual notation is prone to of-by-one errors as you have to either specify the commit before the ones you want to move or count the number of commits.

You may add this rebase-onto function to your ~/.bashrc:

function rebase-onto {
  commit=$(git log --oneline | fzf --prompt 'Select the first commit y...

Git shortcut to fixup a recent commit

git --fixup is very handy to amend a change to a previous commit. You can then autosquash your commits with git rebase -i --autosquash and git will do the magic for you and bring them in the right order. However, as git --fixup wants a ref to another commit, it is quite annoying to use since you always have to look up the sha of the commit you want to amend first.

Inspired by the [shortcut to checkout recent branches with fzf](https://makandracards.com/makandra/505126-g...

NVM: How to automatically switch version when changing directories

The Node Version Manager allows installing multiple NodeJS versions and switching between them.
By default, it does not automatically switch versions when entering a directory that holds a .nvmrc file.

The project's readme document offers a bash function which calls nvm use after each cd. In fact, it replaces cd in your bash.

I did not want to do that, but instead use the $PROMPT_COMMAND feature. So here is my take on it.
Note that it is much shorter, it probably does a f...

How to generate and test a htpasswd password hash

Generate a password

htpasswd -Bn firstname.lastname

This will ask you for a password and use bcrypt (-B, more secure) and print the output to stdout (-n).

Check if password matches the hash

You'll first have to write the password hash to a file:

echo firstname.lastname:$2y$05$4JXxd2GM/J2...9c3KJmFS > htpass_test

Check, if it is correct:

htpasswd -v htpass_test firstname.lastname

You probably should not use the -b switch to read the password from the command line as the password will then be visible...

Delivering Carrierwave attachments to authorized users only

Preparation

To attach files to your records, you will need a new database column representing the filename of the file. To do this, add a new migration (rails g migration <name>) with the following content:

class AddAttachmentToNotes < ActiveRecord::Migration[6.0]
  def change
    add_column :notes, :attachment, :string
  end
end

Don't forget to rename the class and change the column details to fit your purpose. Run it.

1) Deliver attachments through Rails

The first way is to store your Carrierwave attachments not ...

The Ruby Object Model

In Ruby (almost) everything is an Object. While this enables a lot of powerful features, this concept might be confusing for developers who have been programming in more static languages, such as Java or C#. This card should help understanding the basic concepts of Ruby's object model and how things behave.


Usage of objects in Ruby

When working with objects in Ruby, you might think of a "container" that holds metadata, variables and methods. Metadata describes stuff like the object's class or its object_id whi...