In modern default RSpec configurations, your tests are usually run in random order. This helps to detect "flickering" tests that only fail when run in a certain order.

Rails Long Term Support

Rails LTS provides security patches for old versions of Ruby on Rails (2.3, 3.2, 4.2 and 5.2)

• Prevents you from data breaches and liability risks
• Upgrade at your own pace
• Works with modern Rubies

Read more Show archive.org snapshot

The reason for this are tests that have side effects causing other tests to fail later. The hard part is to find the offending test.

Enter rspec --bisect:

1. Say you have a flickering test that passes on its own, but you just saw it fail in a full test run. At the top of the RSpec output, you will see a message like Randomized with seed 12345. Take a note of the number. If this message is missing, your tests are not randomized. You can still use this method, simply skip the --seed.

2. Check if the problem is reproducible by running

   rspec --seed 12345
   

   If the failure does not occur again, your test might instead have some random component to it. rspec --bisect cannot help you with that.

3. Now run

   rspec --bisect --seed 12345
   

   This will take some time (maybe about 2x the usual run duration). If successful, RSpec should give you a minimal failing test suite at the end, like this:

   The minimal reproduction command is:
   rspec './spec/other_spec:20' './spec/flickering_spec:30' --seed 12345
   

   With this info you should be able to find the issue much quicker.

RSpec 3.8 & 3.9 have some bugs where rspec --bisect=verbose --seed 12345 gets stuck after the first spec run (at least 3.9.1 was not fixed despite the promising Changelog that addresses issues with bisect).

Info

I also had the described problem with RSpec 3.10. After upgrading to RSpec 3.11 it was working like expected.