Posted about 2 years ago. Visible to the public. Repeats. Linked content.

RSpec: Debug flickering test suites with rspec --bisect

In modern default RSpec configurations, your tests are usually run in random order. This helps to detect "flickering" tests that only fail when run in a certain order.

The reason for this are tests that have side effects causing other tests to fail later. The hard part is to find the offending test.

Enter rspec --bisect:

  1. Say you have a flickering test that passes on its own, but you just saw it fail in a full test run. At the top of the RSpec output, you will see a message like Randomized with seed 12345. Take a note of the number. If this message is missing, your tests are not randomized. You can still use this method, simply skip the --seed.
  2. Check if the problem is reproducible by running

    Copy
    rspec --seed 12345

    If the failure does not occur again, your test might instead have some random component to it. rspec --bisect cannot help you with that.

  3. Now run

    Copy
    rspec --bisect --seed 12345

    This will take some time (maybe about 2x the usual run duration). If successful, RSpec should give you a minimal failing test suite at the end, like this:

    Copy
    The minimal reproduction command is: rspec './spec/other_spec:20' './spec/flickering_spec:30' --seed 12345

    With this info you should be able to find the issue much quicker.

Once an application no longer requires constant development, it needs periodic maintenance for stable and secure operation. makandra offers monthly maintenance contracts that let you focus on your business while we make sure the lights stay on.

Owner of this card:

Avatar
Tobias Kraze
Last edit:
about 2 years ago
by Arne Hartherz
About this deck:
We are makandra and do test-driven, agile Ruby on Rails software development.
License for source code
Posted by Tobias Kraze to makandra dev
This website uses cookies to improve usability and analyze traffic.
Accept or learn more