Ruby: Generating and parsing JSON, or: understanding JSON::ParserError "unexpected token"

json is part of the standard library of Ruby and deals with JSON, obviously. As you know, JSON is the string format that represents simple data structures. Ruby data structures that resemble Javascript objects can be serialized to JSON with #to_json. These can be restored from a JSON string with JSON.parse().

So what could go wrong here?

JSON.parse("a".to_json)

It will raise JSON::ParserError (784: unexpected token at '"a"'). But why?

Generating JSON vs serializing objects

J...

Parsing JSON with edge cases

The linked article shows that there are unclear parts in the JSON specification and that different parsers treat them differently (which could lead to security vulnerabilities in certain cases).

I was curious what Ruby does (Ruby 2.6.6 with gem json 2.3.0, implementing RFC 7159):

Duplicate Keys

irb(main):001:0> require 'json'
=> true
irb(main):002:0> JSON.parse('{"qty": 1, "qty": -1}')
=> {"qty"=>-1}

Character Collision

irb(main):009:0> JSON.parse('{"qty": 1, "qty\ud800": -1}')
JSON::ParserError (487: incomplete sur...

Webmock < 3.12.1 cannot handle IPv6 addresses correctly

We had the issue, that a VCR spec failed, after updating CarrierWave from version 0.11.0 to 1.3.2.
In this version, CarrierWave uses the gem SsrfFilter, which retrieves the IP addresses for the given hostname and replaces the hostname in the requested url with one of them.

It works with IPv4 addresses, but not with IPv6 addresses, because WebMock cannot handle those correctly:

uri = "#{protocol}://...

Too many parallel test processes may amplify flaky tests

By default parallel_tests will spawn as many test processes as you have CPUs. If you have issues with flaky tests, reducing the number of parallel processes may help.

Important

Flaky test suites can and should be fixed. This card is only relevant if you need to run a flaky test suite that you cannot fix for some reason. If you have no issues...

How to fix: WrongScopeError when using rspec_rails with Rails 6.1

tl;dr: Upgrade the gem to at least 4.0.1

When you use rspec_rails in a version < 4 with Rails 6.1 you may encounter an error like this:

Failure/Error:
  raise WrongScopeError,
    "`#{name}` is not available from within an example (e.g. an " \
    "`it` block) or from constructs that run in the scope of an " \
    "example (e.g. `before`, `let`, etc). It is only available " \
    "on an example group (e.g. a `describe` or `context` block)."
    `name` is not available from within an example (e.g. an `it` block) or from constructs that...

What is a reduction and why Fibers are the answer for Ruby concurrency | julik live

The linked article provides a good overview of the various concurrency primitives in Ruby, and what's changing in Ruby 3.

How to list updateable dependencies with Bundler and Yarn

Bundler

bundle outdated [--filter-major|--filter-minor|--filter-patch]

Example output for bundle outdated --filter-major

Image

Other examples

A useful flag is --strict as it will only list versions that are allowed by your Gemfile requirements (e.g. does not show rails update to 6 if your Gemfile has the line gem 'rails', '~>5.2').

I also experienced that doing upgrades per group (test, development) are easier to do. Thus --groups might also be helpful.

$ bundle...

Some tips for upgrading Bootstrap from 3 to 4

Recently I made an upgrade from Bootstrap 3 to Bootstrap 4 in a bigger project. Here are some tips how to plan and perform such an upgrade. The effort will scale with the size of the project and its structure. If your stylesheets already follow strict rules, it may take less time to adapt them to the new version.

Preparation

There are several gems and libraries that works well with bootstrap or provide at least stylesheets/plugins to easily integrate the bootstrap theme. But very often they only work with specific version or are no long...

How to fix "Command "webpack" not found"

I just ran into this deployment error after switching from the asset pipeline to webpack:

01:05 deploy:assets:precompile
      01 bundle exec rake assets:precompile
      01 Compiling...
      01 Compilation failed:
      01 yarn run v1.22.5
      01 error Command "webpack" not found.
rake stderr: Nothing written

The problem is not related to the "webpack" dependency. You probably just forgot to add a binstub to run "yarn install":

Add these lines to "bin/ya...

Git: Parsing large diffs as a human

I just finished migrating a project from the Asset Pipeline to Webpacker, this is what my diff to master looks like:

5.825 files changed, 44.805 insertions(+), 529.948 deletions(-)
warning: inexact rename detection was skipped due to too many files.
warning: you may want to set your diff.renameLimit variable to at least 5134 and retry the command.

There is no way me or my peer reviewer is able to parse 500k+ lines of code. Fortunately, git has ...

Ruby: How to load a file with a known encoding

In case Ruby does not detected the expected encoding of a file automatically you can specify the known encoding manually.

Example with File.open

file = File.open('some.bin', encoding: Encoding::ASCII_8BIT)
text = file.read
text.encoding => #<Encoding:ASCII-8BIT>

Example with File.read

text = File.read('some.bin', encoding: Encoding::ASCII_8BIT)
text.encoding => #<Encoding:ASCII-8BIT>

More details about the encoding of strings in Ruby can be found [here](https://makandracards.com/makandra/474671-guide-to-string-encodi...

How to communicate between processes in Ruby with sockets

In Ruby you can communicate between processes with sockets. This might be helpful in tests that validate parallel executions or custom finalization logic after the garbage collector. Here is an example how such an communication will look like:

require 'socket'
BUFFER_SIZE = 1024

# DGRAM has the advantage that it stops reading the pipe if the next messages starts. In case the message size is larger than the
# BUFFER_SIZE, you need to handle if you are reading another part of the current message or if you already reading the
# next mess...

Using the Truemail gem to validate e-mail addresses

The Truemail gem (not to be confused with truemail.io) allows validating email addresses, e.g. when users enter them into a sign-up form. It runs inside your application and does not depend on an external SaaS service.

Truemail supports different validation "layers":

  1. Regex validation: if the given address is syntactically valid
  2. DNS validation (called MX validation): if the given domain exists and can receive email
  3. SMTP validation: connects to the host received from DNS and starts a test d...

ExceptionNotification: Fix DNS lookup before plugins call external APIs

The ExceptionNotification has plugins that talk to external APIs rather then just sends emails, like microsoft teams or slack. You might encounter that no exceptions are delivered, which can be dangerous.

To prevent getting into trouble, simply add require 'resolv-replace' on top of your to your initializers/exception_notification.rb file:

require 'resolv-replace'
require 'exception_notification/rails'

ExceptionNotification.configure do |config|
  ...
end

This calls the Ruby DNS resolver before configuring `ExceptionNot...

Call original method when monkey patching

Ruby offers monkey patching methods in order to change the behavior of a library if there's no better way.

We can call the method we're overriding inside our monkey patch:

class Foo
  def bar(argument)
    'Hello' + argument
  end
end 

module FooExtensions
  def bar
    super(' in my') + ' World'
  end
end

class Foo
  prepend FooExtensions # the only change to above: prepend instead of include
end

Foo.new.bar # => 'Hello in my...

Ruby: How to keep split delimiter (separate, or as part of substrings)

Ruby's String#split returns an array of substrings from the given string. Usually, this is missing the split characters:

>> 'user@example.com'.split('@')
=> ["user", "example.com"]

If you want to join those parts later on, you might know the split character and can just use it to join explicitly.
But if you split by a regular expression (for a set of split characters) that information is lost:

>> 'user@example.com'.split(/[@\.]/)
=> ["user", "example", "com"]

You can use a capture group to make those characters ...

SSHKit 1.9.0 failure for Capistrano deploy

SSHKit 1.9.0 might fail with the following error, when trying to deploy a Rail application. Upgrading the gem to version 1.21.0 fixed the issue.

Traceback (most recent call last):
	17: from /home/user/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/sshkit-1.9.0/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
	16: from /home/user/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/sshkit-1.9.0/lib/sshkit/backends/abstract.rb:29:in `run'
	15: from /home/user/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/sshkit-1.9....

Missing certificates for rubygems and bundler in Ruby 1.8.7

Using Ruby 1.8.7 you will not be able to use the maximum versions Rubygems 1.8.30 and Bundler 1.17.3 with https://rubygems.org/ anymore. This is a result of a server certificate on December 5th, 2020. The resulting errors will look like following:

  • TypeError: can't modify frozen object
  • Could not verify the SSL certificate for https://rubygems.org/*
  • Bundler::Fetcher::CertificateFailureError: Could not verify the SSL certificate for https://index.rubygems.org/versions.
  • `Error fetching data: hostname was not m...

Configuring Webpacker deployments with Capistrano

When deploying a Rails application that is using Webpacker and Capistrano, there are a few configuration tweaks that optimize the experience.

Using capistrano-rails

capistrano-rails is a Gem that adds Rails specifics to Capistrano, i.e. support for Bundler, assets, and migrations. While it is designed for Asset Pipeline (Sprockets) assets, it can easily be configured for Webpacker. This brings these features to the Webpacker world:

  • Automatic removal of expired assets
  • Manifest backups

How to fix: Pasting in IRB 1.2+ is very slow

IRB 1.2 (shipped with Ruby 2.7, but works on 2.5+) brings pretty syntax highlighting and multiline cursor navigation. However, pasting longer contents is incredibly slow. You can fix that by disabling said features. [1]

Ruby 3.0.0-pre2 solved the issue (however, the fix does not appear to be included in IRB 1.2.6, it must be Ruby itself).

Option 1:

Add a command line flag when opening an IRB:

irb --nomultiline

This also works on modern Rails...

Rails developers: Have better context in Git diffs

Git diffs show the surrounding contexts for diff hunks. It does so by applying regular expressions to find the beginning of a context. When it comes to Ruby, however, it will not find method heads and travel up to the class definition:

@@ -24,7 +24,7 @@ class TicketPdf # <=== Actually expected here: the method definition
     ApplicationController.render(
       "tickets/index.html.haml",
       layout: "tickets",
-      assigns: { tickets: tickets }
+      assigns: { tickets: tickets, event_name: event_name }
     )
   end
 end
```...

Show/Hide Rubocop marking in RubyMine

If you have installed Rubocop in your project, RubyMine can show you Rubocop violations immediately in your editor. You probably already know this feature.

Example

Image

Enable/Disable marking

If your RubyMine does not show you any violations, although there are some, you may have to enable the setting first.

To do so, open Navigate -> Search Everywhere -> Actions (Or use the shortcut CTRL + SHIFT + A) and type in "rubocop", then you should see some...

Convert curl commands to ruby code

curl-to-ruby is a handy tool that converts your curl command to ruby code that uses the Net::HTTP library.

Example

curl -X POST -d
  "grant_type=password&email=email&password=password"
  localhost:3000/oauth/token

will output to:

require 'net/http'
require 'uri'

uri = URI.parse("http://localhost:3000/oauth/token")
request = Net::HTTP::Post.new(uri)
request.set_form_data(
  "email" => "email",
  "grant_type" => "password",
  "password" => "password",
)

req_options =...

Ruby: Comparing a string or regex with another string

In Rubocop you might notice the cop Style/CaseEquality for e.g. this example:

def foo(expected, actual)
  expected === actual
end

In case expected is a Regex, it suggests to change it to the following pattern:

def foo(expected, actual)
  expected.match?(actual)
end

In case expected is a Regex or a String, you need to keep ===. Otherwise the actual expression is always converted to a regular expression.

# For expected === actual
foo('Test(s)', 'Test(s)') #=> true

# For expected.match?(actual)
foo('Test(...