Shell script to magically configure display setup

Here is a bash script that I use to auto-configure displays on Ubuntu 24.04 with Xorg.

Background

  • Ubuntu always sets the primary display to the 1st (i.e. internal) display whenever I connect to a new Dock/Hub.
    • I want my primary display to be the large display.
    • My notebook is always placed left of external displays, so the 2nd display will be the center (or only) external display and should be primary.
  • I also want all my displays to be placed horizontally, but bottom-aligned (the default would be aligned at their top edges)....

Debugging Capistrano

Capistrano 3 has a doctor task that will print information about

  • Environment: Ruby, Rubygems and Bundler versions
  • List of Capistrano gems and whether an update is available
  • All config variables and their values
  • Capistrano server config
$ bundle exec cap staging doctor

Ruby: Retrieving and processing files via Selenium and JavaScript

This card shows an uncommon way to retrieve a file using selenium where JavaScript is used to return a binary data array to Ruby code.

The following code example retrieves a PDF but the approach also works for other file types.

require "selenium-webdriver"

selenium_driver = Selenium::WebDriver.for :chrome
selenium_driver.navigate.to('https://example.com')
link_to_pdf = 'https://blobs.example.com/random-pdf'

binary_data_array = selenium_driver.execute_script(<<-JS, link_to_pdf)
  const response = await fetch(arguments[0])

  if (!r...

RSpec: Applying stubs only within a block

When you mocked method calls in RSpec, they are mocked until the end of a spec, or until you explicitly release them.

You can use RSpec::Mocks.with_temporary_scope to have all mocks applied inside a block to be released when the block ends.
Example:

RSpec::Mocks.with_temporary_scope do
  allow(Rails).to receive(:env).and_return('production'.inquiry)
  puts Rails.env # prints "production"
end
puts Rails.env # prints "test"

Note that, when overriding pre-existing mocks inside the block, they are not reverted to the previously ...

Best practices: Writing a Rails script (and how to test it)

A Rails script lives in lib/scripts and is run with bin/rails runner lib/scripts/.... They are a simple tool to perform some one-time actions on your Rails application. A Rails script has a few advantages over pasting some prepared code into a Rails console:

  • Version control
  • Part of the repository, so you can build on previous scripts for a similar task
  • You can have tests (see below)

Although not part of the application, your script is code and should adhere to the common quality standards (e.g. no spaghetti code). However, a script...

Bash: How to count and sort requests by IP from the access logs

Example

87.140.79.42 - - [23/Jan/2024:09:00:46 +0100] "GET /monitoring/pings/ HTTP/1.1" 200 814 "-" "Ruby"
87.140.79.42 - - [23/Jan/2024:09:00:46 +0100] "GET /monitoring/pings/ HTTP/1.1" 200 814 "-" "Ruby"
87.140.79.41 - - [23/Jan/2024:09:00:46 +0100] "GET /monitoring/pings/ HTTP/1.1" 200 814 "-" "Ruby"
87.140.79.42 - - [23/Jan/2024:09:00:46 +0100] "GET /monitoring/pings/ HTTP/1.1" 200 814 "-" "Ruby"

Goal

Count and sort the number of requests for a single IP address.

Bash Command

awk '{ print $1}' test.log | sort...

Bash: How to grep logs for a pattern and expand it to the full request

Example

I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [4cdad7a4-8617-4bc9-84e9-c40364eea2e4] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] test
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [6e047fb3-05df-4df7-808e-efa9fcd05f87] more
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- : [53a240c1-489e-4936-bbeb-d6f77284cf38] nope
I, [2024-01-21T06:22:17.484221 #2698200]  INFO -- ...

Opening a zipped coverage report with one click

Image

Tested on Ubunut 22.04

1. Opener script

  • Create a file ~/.local/bin/coverage_zip_opener with:
#!/bin/bash

tmp_folder="/tmp/coverage-report-opener"

if [ -z "$1" ]
then
 echo "Usage: coverage_zip_opener [filename]"
 exit -1
fi

if ! [[ "$1" =~ ^.*Pipeline.*Coverage.*\.zip$ || "$1" =~ ^.*merged_coverage_report.*\.zip$ ]]; then
 file-roller "$1"
 exit 0
fi

rm -Rf $tmp_folder

unzip -qq "$1" -d $tmp_folder
index_filename=$(find /tmp/coverage-report-opener -name "index.html" | ...

open-next-failure: An alias to speed up test debugging

Getting an entire test suite green can be a tedious task which involves frequent switches between the CLI that is running tests back to the IDE where its cause can be fixed.

The following bash aliases helped me speed up that process:

alias show-next-failure="bundle exec rspec --next-failure"
alias open-next-failure="show-next-failure || show-next-failure --format json  | jq -r '.examples[0]' | jq '\"--line \" + (.line_number|tostring) + \" \" + .file_path' | xargs echo | xargs rubymine"

There is a lot going on above but the gist...

Capistrano 3: Running a command on all servers

This Capistrano task runs a command on all servers.

bundle exec cap production app:run cmd='zgrep -P "..." RAILS_ROOT/log/production.log'

Code

# lib/capistrano/tasks/app.rake

namespace :app do

  # Use e.g. to grep logs on all servers:
  #   b cap production app:run_cmd cmd='zgrep -P "..." RAILS_ROOT/log/production.log' 
  #
  # * Use RAILS_ROOT as a placeholder for the remote Rails root directory.
  # * Append ` || test $? =1;` to grep calls in order to avoid exit code 1 (= "nothing found")
  # * To be able to process ...

Debug MiniMagick calls in your Rails app

Most of our applications use CarrierWave for file uploads. CarrierWave has an integrated processing mechanism for different file versions with support for ImageMagick through CarrierWave::MiniMagick (which requires the mini_magick gem). In case your processing runs into an error, CarrierWave will just swallow it and rethrow an error with a very generic message like Processing failed. Maybe it is not an image? which does not help you finding out what the actual problem is. CarrierWave probably does this for security purposes, but does n...

How to make sure that manual deploy tasks (scheduled in Pivotal Tracker) are executed on deploy (with Capistrano)

We regularly have tasks that need to be performed around a deploy. Be it to notify operations about changed application behavior, be it to run a little oneline script after the deploy. Most database-related stuff can be handled by migrations, but every once in a while, we have tasks that are much easier to be performed manually.

Writing deploy tasks

Here is how we manage the deploy tasks themselves:

  • Deploy tasks are written inside the Pivotal Tracker story description, clearly marked (e.g. with a headline "Deploy task")
  • We disting...

CarrierWave: Processing images with libvips

When you write your next CarrierWave uploader, consider processing your images with libvips instead of ImageMagick.

Reasons for libvips

There are several upsides to using libvips over ImageMagick:

How to pretty print all values in a Redis database

With this Ruby script you can print all values in a Redis database to your console (derived from this bash script).

Note: Do not run this command in production. This is for debugging purposes only.

def pretty_print_redis(redis)
  redis.keys.each_with_object({}) do |key, hash|
    type = redis.type(key)

    hash[key] = case type
    when 'string'
      redis.get(key)
    when 'hash'
      redis.hgetall(key)
    when 'list'
      redis.lrange(key, 0, -1)
    when 'set'
      redis.smembers(...

Sidekiq 7: Rate limiting with capsules

Sidekiq 7 adds a new feature called capsules.

Use cases:

  • a chrome queue limited to 1 for e.g. PDF processing to not overload the application server
  • an api queue, that limits a queue to 2 to protect the API server from too many requests in parallel

Example:

Sidekiq.configure_server do |config|
  # Edits the default capsule
  config.queues = %w[critical default low]
  config.concurrency = 5

  # Define a new capsule which ...

Issue Checklist Template

This is a checklist I use to work on issues. For this purpose I extracted several cards related to the makandra process and ported them into a check list and refined that over time a little bit.

This task list is divided by the Gate keeping process in the following steps:

1. Starting a new feature
2. Working on the issue
3. Finishing a feature
4. After Review

Here are some ti...

Git: Restore

tl;dr

git checkout is the swiss army of git commands. If you prefer a semantically more meaningful command for restoring tasks, use git restore instead.

With this command you can ...

  • ... do unstaging - git restore --staged
  • ... discard staged changes - git restore --staged --worktree
  • ... discard unstaged changes - git restore
  • ... restore deleted files - git restore
  • ... restore historic versions - git restore --source
  • ... recreate merge conflicts - git restore --merge
  • ... specifiy...

How to fix "Exit with code 1 due to network error: ProtocolUnknownError" with wkhtmltopdf

New versions of wkhtmltopdf dissallow file:// URLs by default. You can allow them by passing --enable-local-file-access.

If you are using PDFKit, set the option

PDFKit.configure do |config|
  config.default_options = {
    enable_local_file_access: true,
  }
end

This will be necessary in many setups to allow wkhtmltopdf to fetch assets (such as stylesheets) from the filesystem.

Note on security

Allowing this poses some risk when you render user input, since it might be feasible to include data from the local filesyste...

Converting SVG to other vector formats without Inkscape

If you need to convert an SVG source to PS or EPS, the most common suggestion on the interwebs is to use Inkscape from the commandline.
Inkscape is a fairly resource-heavy tool with lots of dependencies. A great alternative for converting is CairoSVG.

CairoSVG is available on most Linux distros through their package management systems, e.g. apt install cairosvg on Ubuntu.
It has few dependencies (most importantly Python 3 and some related packages, but really not much)...

How to debug issues with zeitwerk and Rails

In case you have trouble with the zeitwerk autoloader, you can check out the documentation Autoloading and Reloading Constants and Classic to Zeitwerk HOWTO for some debugging hints.

For myself it was useful to print the registered constants and the file references during the boot. Therefore you need to add Rails.autoloaders.log! at the end of your config/application.rb file. You could also run `bin...

How to get information about a gem (via CLI or at runtime from Ruby)

When you need information about a gem (like version(s) or install path(s)), you can use the gem binary from the command line, or the Gem API inside a ruby process at runtime.

gem binary (in a terminal)

You can get some information about a gem by running gem info <gem name> in your terminal.

Example:

$ gem info irb

*** LOCAL GEMS ***

irb (1.4.1, 1.3.5)
    Author: Keiju ISHITSUKA
    Homepage: https://github.com/ruby/irb
    Licenses: Ruby, BSD-2-Clause
    Installed at (1.4.1): /home/arne/.rbenv/versions/3.0.3/lib/ruby/g...

Generating an Entity Relationship Diagram for your Rails application

This card explains how to generate an entity relationship diagram for your Rails application.
We also show how to limit your ERD to a subset of models, e.g. models inside a namespace.

Generating a full ERD

Option A: RubyMine

  1. Right-click anywhere in your project tree
  2. In the context menu, find the "Diagrams" menu item at/near the bottom
  3. Inside, choose "Show diagram" → "Rails Model Dependency Diagram"
  4. A new tab will open with the diagram inside. You can modify it there, and export it as an image.

Option B: Use rails-e...

Using feature flags to stabilize flaky E2E tests

A flaky test is a test that is often green, but sometimes red. It may only fail on some PCs, or only when the entire test suite is run.

There are many causes for flaky tests. This card focuses on a specific class of feature with heavy side effects, mostly on on the UI. Features like the following can amplify your flakiness issues by unexpectedly changing elements, causing excessive requests or other timing issues:

  • Lazy loading images
  • Autocomplete in search f...

Git shortcut to rebase onto another branch

Inspired by recent "git shortcut" cards I figured it would be nice to have one of these for rebasing a few commits onto another branch. The usual notation is prone to of-by-one errors as you have to either specify the commit before the ones you want to move or count the number of commits.

You may add this rebase-onto function to your ~/.bashrc:

function rebase-onto {
  commit=$(git log --oneline | fzf --prompt 'Select the first commit y...