Read more

How to make Webpacker compile once for parallel tests, and only if necessary

Arne Hartherz
June 30, 2017Software engineer at makandra GmbH

Webpack is the future. We're using it in our latest Rails applications.

Illustration online protection

Rails Long Term Support

Rails LTS provides security patches for old versions of Ruby on Rails (2.3, 3.2, 4.2 and 5.2)

  • Prevents you from data breaches and liability risks
  • Upgrade at your own pace
  • Works with modern Rubies
Read more Show archive.org snapshot

For tests, we want to compile assets like for production.
For parallel tests, we want to avoid 8 workers compiling the same files at the same time.
When assets did not change, we do not want to spend time compiling them.

Here is our solution for all that.

Its concept should work for all test suites.

Copy the following to config/initializers/webpacker_compile_once.rb. It will patch Webpacker, but only for the test environment:

# Avoid hardcoded asset hosts in webpack, otherwise all chunks would be loaded through the
# Capybara server of the first test process. If that test process hasn't launched a Capybara
# server yet, or if it finishes before other processes, chunk loading in other processes will
# fail with a message like "ChunkLoadError: Loading chunk 0 failed".
ENV['WEBPACKER_ASSET_HOST'] = ''

module WebpackerCompileOnce
  COMPILE_WAIT_TIMEOUT = 180
  COMPILE_WAIT_POLL_INTERVAL = 0.3

  def compile
    # Calling #compile will at least once compute SHAs over the content
    # of all files in app/. Files cannot change in tests, so we immediately
    # return if we have compiled before.
    if test?
      if @compiled_before
        return true
      else
        @compiled_before = true
      end
    end

    if parallel_tests?
      if parallel_tests_number == 0
        # Before compiling Webpacker::Compiler will check whether files in app/ and
        # other places changed.
        super
      else
        # Wait until the first parallel_tests process has finished compilation.
        wait_until_fresh
        true
      end
    else
      # Development, production, non-parallel test run.
      super
    end
  end

  def fresh?
    # Stock Webpacker checks if JavaScript changed in app, yarn.lock or config/webpack.
    # We want to be a bit stricter about freshness than stock Webpacker:
    # We *do* want to compile if someone manually emptied the packs-test folder in an attempt
    # to fix a broken build.
    super && Dir.exist?(public_output_path) && !Dir.empty?(public_output_path)
  end

  delegate :public_output_path, to: :config

  private

  def test?
    Rails.env.test?
  end

  def parallel_tests?
    test? && !!parallel_tests_number
  end

  def parallel_tests_number
    if (raw = ENV['TEST_ENV_NUMBER'])
      raw.to_i
    end
  end

  def caching_watched_files_digest(&block)
    @cache_watched_files_digest = true
    block.call
  ensure
    @cache_watched_files_digest = false
  end

  def watched_files_digest
    if @cache_watched_files_digest
      @previous_watched_files_digest ||= super
    else
      super
    end
  end

  # Waits until the first parallel_tests process has finished compiling.
  def wait_until_fresh
    # We're going to call #fresh? a lot. This will call #watched_files_digest, which
    # computes SHAs over the contents of all files in app/ and other places.
    # Because this is expensive we do it only once for the duration of this method.
    caching_watched_files_digest do
      elapsed = 0
      loop do
        break if fresh?

        sleep(COMPILE_WAIT_POLL_INTERVAL)

        elapsed += COMPILE_WAIT_POLL_INTERVAL
        if elapsed > COMPILE_WAIT_TIMEOUT
          raise "First parallel_tests process did not compile within #{COMPILE_WAIT_TIMEOUT} seconds. It may have crashed?"
        end
      end
    end
  end

end

Webpacker::Compiler.prepend(WebpackerCompileOnce)

It's a good idea to compile your Webpacker assets once before your end-to-end integration test suite. Otherwise the first rendered javascript_pack_tag would start compilation, causing the helper to freeze for a minute and current Capybara visit to timeeout.

To compile assets before all Cucumber scenarios, copy the following to features/support/webpacker.rb:

# Compile before the test suite starts.
# Otherwise the first rendered javascript_pack_tag would start compilation,
# causing the current Capybara visit to time out.
Webpacker.compile

That's it. Enjoy! 🎉


Similar scenario: running an application with Webpacker on multiple servers. How to serve identical assets from all servers.

Arne Hartherz
June 30, 2017Software engineer at makandra GmbH
Posted by Arne Hartherz to makandra dev (2017-06-30 20:00)