RSpec: Debug flickering test suites with rspec --bisect

In modern default RSpec configurations, your tests are usually run in random order. This helps to detect "flickering" tests that only fail when run in a certain order.

The reason for this are tests that have side effects causing other tests to fail later. The hard part is to find the offending test.

Enter rspec --bisect:

1. Say you have a flickering test that passes on its own, but you just saw it fail in a full test run. At the top of the RSpec output, you will see a message like Randomized with seed 12345. Take a note of the number. If this message is missing, your tests are not randomized. You can still use this method, simply skip the --seed.

2. Check if the problem is reproducible by running

   Copyrspec --seed 12345
   

   If the failure does not occur again, your test might instead have some random component to it. rspec --bisect cannot help you with that.

3. Now run

   Copyrspec --bisect --seed 12345
   

   This will take some time (maybe about 2x the usual run duration). If successful, RSpec should give you a minimal failing test suite at the end, like this:

   CopyThe minimal reproduction command is:
   rspec './spec/other_spec:20' './spec/flickering_spec:30' --seed 12345
   

   With this info you should be able to find the issue much quicker.

RSpec 3.8 & 3.9 have some bugs where rspec --bisect=verbose --seed 12345 gets stuck after the first spec run (at least 3.9.1 was not fixed despite the promising Changelog that addresses issues with bisect).

Info

I also had the described problem with RSpec 3.10. After upgrading to RSpec 3.11 it was working like expected.