RSpec: Debug flickering test suites with rspec --bisect

In modern default RSpec configurations, your tests are usually run in random order. This helps to detect "flickering" tests that only fail when run in a certain order.

The reason for this are tests that have side effects causing other tests to fail later. The hard part is to find the offending test.

Enter rspec --bisect:

Say you have a flickering test that passes on its own, but you just saw it fail in a full test run. At the top of the RSpec output, you will see a message like Randomized with seed 12345. Take a note of the number. If this message is missing, your tests are not randomized. You can still use this method, simply skip the --seed.
Check if the problem is reproducible by running
```
rspec --seed 12345
```
If the failure does not occur again, your test might instead have some random component to it. rspec --bisect cannot help you with that.
Now run
```
rspec --bisect --seed 12345
```
This will take some time (maybe about 2x the usual run duration). If successful, RSpec should give you a minimal failing test suite at the end, like this:
```
The minimal reproduction command is:
rspec './spec/other_spec:20' './spec/flickering_spec:30' --seed 12345
```
With this info you should be able to find the issue much quicker.

RSpec 3.8 & 3.9 have some bugs where rspec --bisect=verbose --seed 12345 gets stuck after the first spec run (at least 3.9.1 was not fixed despite the promising Changelog that addresses issues with bisect).

Info

I also had the described problem with RSpec 3.10. After upgrading to RSpec 3.11 it was working like expected.

rspec.info

Tobias Kraze

makandra.de

Say thanks9

Last edit

2022-04-25

Julian

License

Source code in this card is licensed under the MIT License.

How to: Use git bisect to find bugs and regressions

Git allows you to do a binary search across commits to hunt down the commit that introduced a bug.

Given you are currently on your branch's HEAD that is not working as expected, an example workflow could be:

git bisect start # Start bisectin...

How to fix: WrongScopeError when using rspec_rails with Rails 6.1

tl;dr: Upgrade the gem to at least 4.0.1

When you use rspec_rails in a version < 4 with Rails 6.1 you may encounter an error like this:

Failure/Error:
  raise WrongScopeError,
    "`#{name}` is not available from within an example (e.g. an ...

Posted by Tobias Kraze to makandra dev (2017-05-12 09:37)

RSpec: Debug flickering test suites with rspec --bisect

Info

Related cards:

Recipe for testing any validation