Using the Truemail gem to validate e-mail addresses

The Truemail gem (not to be confused with truemail.io) allows validating email addresses, e.g. when users enter them into a sign-up form. It runs inside your application and does not depend on an external SaaS service.

Truemail supports different validation "layers":

  1. Regex validation: if the given address is syntactically valid
  2. DNS validation (called MX validation): if the given domain exists and can receive email
  3. SMTP validation: connects to the host received from DNS and starts a test d...

Transporting blank values in URL queries

URLs can transport key/value pairs ("parameters") using this syntax:

/path?foo=bar

If the value is blank, mind these subtle differences:

URL Meaning
/path?foo= Parameters have a key foo. Its value is an empty string.
/path?foo Parameters have a key foo. Its value is null.
/path Parameters have no key foo.

Configuring Webpacker deployments with Capistrano

When deploying a Rails application that is using Webpacker and Capistrano, there are a few configuration tweaks that optimize the experience.

Using capistrano-rails

capistrano-rails is a Gem that adds Rails specifics to Capistrano, i.e. support for Bundler, assets, and migrations. While it is designed for Asset Pipeline (Sprockets) assets, it can easily be configured for Webpacker. This brings these features to the Webpacker world:

  • Automatic removal of expired assets
  • Manifest backups

Minidusen: Filtering associated records

Minidusen lets you find text in associated records.

Assume the following model where a Contact record may be associated with a Group record:

class Contact < ApplicationRecord
  belongs_to :group

  validates_presence_of :name, :street, :city, :email
end

class Group < ApplicationRecord
  has_many :contacts

  validates_presence_of :name
end

We can filter contacts by their group name by joining the groups table and filtering on a joined column.
Note how the joined column is qualified as groups.name (rather than just `na...

Passive event listeners may speed up your scroll and touch events

Scroll and touch event listeners tend to be computationally expensive as they are triggered very often. Every time the event is fired, the browser needs to wait for the event to be processed before continuing - the event could prevent the default behavior. Luckily there is a concept called passive event listeners which is supported by all modern browsers.

Below are the key parts quoted from WICG's explainer on passive event listeners. See [this demo video](https://www.youtube.com/watch?v=NPM6172...

How to negate scope conditions in Rails

Sometimes you want to find the inverse of an ActiveRecord scope. Depending on what you want to achieve, this is quite easy with Rails 7, and a bit more complicated with Rails 6 and below, or when the inverse scope may contain NULL values. [1]

There are two different ways of "inverting a scope":

As an example, consider the following model.

class User < ApplicationRecord
  scope :admins, -> { where(role: ['admin', 'superuser']) }
  # ...
end

Mathematical NOT

You know this one from basic set theory. It proces the "complementa...

Manage Linux services on the command line (Ubuntu)

Ubuntu 18.04 uses systemd to manage services.

There are basically two commands for listing all services and manipulating the state of a certain service: service and systemctl:

  • service manages System V init scripts
  • systemctl controls the state of the systemd system and service manager. It is backwards compatible to System V and includes the System V services

Therefore I prefer to use systemctl.


See which services are there

>systemctl list-units -a --type=service
  UNIT                                 LOAD     ...

Migrate gem tests from Travis CI to Github Actions with gemika

We currently test most of our gems on Travis CI, but want to migrate those tests to Github Actions. This is a step-by-step guide on how to do this.

Note that this guide requires the gem to use gemika.

  1. Go to a new "ci" branch:
    git checkout -b ci
    
  2. Update gemika to version >= 0.5.0 in all your Gemfiles.
  3. Have gemika generate a Github Actions workflow definition by running
    mkdir -p .github/workflows; bundle exec rake gemika:generate_github_actions_workflow > .github/workf...
    

Debugging SPF records

While debugging a SPF record I found spf-record.de to be very helpful.

  • it lists all IPs that are covered by the SPF record
  • shows syntax errors
  • helps you debugging errors like DNS lookup limit reached
  • it also lets you test a new SPF strings before applying it. This can save you time as you don't have to loop with operations

Also the advanced check at vamsoft.com has a very good interface to test new SPF policies.

PostgreSQL: Importing dumps created with newer versions

When loading a database dump created with pg_dump into your database, you might run into an error like

pg_restore: error: unsupported version (1.15) in file header

This is because your local pg_restore version is too old to match the format created by pg_dump. The version of the PostgreSQL server doesn't matter here.

For example, the official Ubuntu 20.04 sources include only PostgreSQL 12, so your pg_restore version will also be v12. Ubuntu 22.04 includes version 14 in its sources.
Both seem to be incompatible with dumps ...

Rails: How to restore a postgres dump from the past

It sometimes happen that a database dump, that would want to insert into your development database, does not match the current schema of the database. This often happens when you have an old dump, but your current setup is up to date with the the master.

Hint: In most cases it is sufficient to delete and recreate the local database in order to import the dump. If any problems occur, proceed as follows:

1. Figure out the original migration status of the dumpfile

  • Convert your dump to plaintext: `pg_restore -f some.dump > some.dump....

How to implement simple queue limiting/throttling for Sidekiq

The sidekiq-rate-limiter gem allows rate-limiting Sidekiq jobs and works like a charm. However, it needs to be integrated on a per-worker basis.

If you want to limit a whole queue instead, and if your requirements are simple enough, you can do it via a Sidekiq middleware yourself.

Here is an example that limits concurrency of the "mailers" queue to 1. It uses a database mutex via the [with_advisory_lock](https://github.com/ClosureTree/wit...

Rails: How to list all validations on a model or an attribute

If a model inherits from others or uses many concerns / traits, it might be hard to see in the code which validators it has.
But fortunately there's a method for that:

irb(main):002:0> pp UserGroup.validators
[#<ActiveModel::Validations::InclusionValidator:0x00007f55efff97a8
  @attributes=[:deleted],
  @delimiter=[true, false],
  @options={:in=>[true, false], :allow_nil=>false}>,
 #<ActiveModel::Validations::InclusionValidator:0x00007f55f15748d0
  @attributes=[:cancelled],
  @delimiter=[true, false],
  @options={:in=>[true, false], ...

How to include Sidekiq job IDs in Rails logs

When logging in Rails, you can use the log_tags configuration option to add extra information to each line, like :request_id or :subdomain. However, those are only valid inside a request context and have no effect when your application is logging from inside a Sidekiq process.
This includes custom as well as any framework logs, like query logging from ActiveRecord.

Since Sidekiq Workers run inside threads of a single process, running multiple jobs in...

How to generate GIDs from an ActiveRecord scope

ActiveRecord provides the ids method to pluck ids from a scope, but what if you need to pluck Global IDs?

While you could just call map(&:to_global_id) on your scope, this approach would instantiate each record just to do that. When you have many records, this will at the very least be slow.

Here is a method that does it for you efficiently. It respects Single Table Inheritance (STI).
Put it in your project's ApplicationRecord to make it available on all models.

class ApplicationRecord
  ...

Workflow: How to use a key management service to encrypt passwords in the database

This is an extract from the linked article. It shows an approach on how to implement encrypted passwords with the AWS Key Management Service (KMS).

For most applications it's enough to use a hashed password with a salt (e.g. the gem devise defaults to this).

Upon password creation

  1. Generate hash as hash of password + salt.

  2. Encrypt the hash with a public key from KMS (you can store the public key in your server code).

  3. In your database sto...

Carrierwave: How to migrate to another folder structure

A flat folder structure can be cool if you have only a few folders but can be painful for huge amounts. We recently had this issue in a project with more than 100.000 attachments, where we used a structure like this /attachments/123456789/file.pdf.

Even the ls command lasted several minutes to show us the content of the attachments folder.

So we decided to use a more hierarchical structure with a limited maximum of folder per layer. Here are a few tips how to migrate your files to their new...

How to fix: Rails query logs always show lib/active_record/log_subscriber.rb as source

Rails 5.2+ supports "verbose query logs" where it shows the source of a query in the application log.
Normally, it looks like this:

  User Load (0.5ms)  SELECT "users".* FROM "users" WHERE ...
  ↳ app/controllers/users_controller.rb:42:in `load_users'

However, you may encounter ActiveRecord's LogSubscriber as the source for all/most queries which is not helpful at all:

  User Load (0.5ms)  SELECT "users".* FROM "users" WHERE ...
  ↳ activerecord (6.0.3.3) lib/active_record/log_subscriber.rb:100:in `debug'

While th...

Service Worker series by GoMakeThings

Learn how to create offline applications with service workers.

  1. The amazing power of service workers
  2. Writing your first service worker with vanilla JS
  3. Saving recently viewed pages offline with service workers and vanilla JS
  4. Offline first with service workers and vanilla JS
  5. Improving web font performance with service workers
  6. How to set an expiration date for items in a service worker cache
  7. How to update a service worker
  8. How to trigger a service worker function from the front end with vanilla JS
  9. How to immediately ...

Controlling issue grouping in Sentry

When you use Sentry to monitor exceptions, an important feature is Sentry's error grouping mechanism. It will aggregate similar error "events" into one issue, so you can track and monitor it more easily. Grouping is especially important when you try to silence certain errors.

It is worth understanding how Sentry's grouping mechanism works.

The default grouping mechanism

The exact algorithm has changed over time, and Sentry will keep using the algorithm t...

Chrome Lighthouse

Chrome has a built-in utility to check performance and accessibility (and more) of your web app: Lighthouse.

Open the Developer Tools and go to the lighthouse tab:

Image

Then you'll see some suggestions on how to improve your site.
This is cool, because you can even use it with non-public pages or your development environment (but be aware that some settings we're using for development, like not minifying JS and CSS files, might ruin your stats)...

Rails: How to get the ordered list of used middlewares

Rails middlewares are small code pieces that wrap requests to the application. The first middleware gets passed the request, invokes the next, and so on. Finally, the application is invoked, builds a response and passes it back to the last middleware. Each middleware now returns the response until the request is answered. Think of it like Russian Dolls, where each middleware is a doll and the application is the innermost item.

You can run rake middleware to get the ordered list of used middlewares in a Rails application:

$> rake midd...

PostgreSQL: How to use with_advisory_lock to prevent race conditions

If you want to prevent that two processes run some code at the same time you can use the gem with_advisory_lock.

What happens

  1. The thread will wait indefinitely until the lock is acquired.
  2. While inside the block, you will exclusively own the advisory lock.
  3. The lock will be released after your block ends, even if an exception is raised in the block.

This is usually required if there is no suitable database row to lock on.

Example

You want to generate a...

How to cycle through grep results with vim

grep is the go-to CLI tool to accomplish tasks like filtering large files for arbitrary keywords. When additional context is needed for search results, you might find yourself adding flags like -B5 -A10 to your query. Now, every search result covers 16 lines of your bash.

There is another way: You can easily pipe your search results to the VIM editor and cycle through them.

Example: Searching for local occurrences of "User"

vim -q <(grep -Hn -r "User" .)

# vim -q starts vim in the "quickfix" mode. See ":help quickfix"
# grep...