Debugging flickering VCR tests

We often use VCR to stub external APIs. Unfortunately VCR can have problems matching requests to recorded cassettes, and these issues are often hard to debug.

VCR's error messages mostly look like this and are not very helpful:

CopyAn HTTP request has been made that VCR does not know how to handle:
         POST http://another-site.de:9605/json/index

VCR fails if the request does not exactly look like the request it has recorded. If the request is different in any way it will assume an error, because it can not know how the system it mocks would answer a request it has not yet recorded.

Some signs an error is caused by randomness

• The test is green on a single run but red when run in parallel
• The test is always green on recording but red when the cassette is used
• The test is green on one day and red the following day

Finding the randomness

Taking a close look at the diff of the yml file that worked before and the one that is recorded in the situation the exisiting one did not work can be helpful in many cases, but the cassette yml is not easy to read.

Follow these steps to make a recorded VCR request more human readable:

1. Search the cassette for the request that fails
2. Copy the request with all parameters
3. Open a rails console, assign the request to a variable and unescape it:

Copymy_project> var = "%5B%7B%22id%22%3A%22NS_test__minze_NS%2Fgid%3A%2F%2Fanother-site%2FPage%3A%3APressKit%2F123%22%2C%22fields%22%3A%7B%22title_de%22%3A%5B%22Press%20kit%2016%22%5D%2C%22title_en%22%3A%5B%22Press%20kit%2016%20(en)%22%5D%2C%22body_de%22%3A%5B%22%22%5D%2C%22body_en%22%3A%5B%22%22%5D%2C%22_date.date_de%22%3A%5B%222015-01-01%22%5D%2C%22_date.date_en%22%3A%5B%222015-01-01%22%5D%2C%22""%5B%7B%22id%22%3A%22NS_test__minze_NS%2Fgid%3A%2F%2Fmy-site%2FPage%3A%3APressKit%2F123%22%2C%22fields%22%3A%7B%22title_de%22%3A%5B%22Press%20kit%2016%22%5D%2C%22title_en%22%3A%5B%22Press%20kit%2016%20(en)%22%5D%2C%22body_de%22%3A%5B%22%22%5D%2C%22body_en%22%3A%5B%22%22%5D%2C%22_date.date_de%22%3A%5B%222015-01-01%22%5D%2C%22_date.date_en%22%3A%5B%222015-01-01%22%5D%2C%22"
my_project> puts CGI.unescape(var)
[{"id":"NS_test__minze_NS/gid://my-site/Page::PressKit/123","fields":{"title_de":["Press kit 16"],"title_en":["Press kit 16 (en)"],"body_de":[""],"body_en":[""],"_date.date_de":["2015-01-01"],"_date.date_en":["2015-01-01"],"

Now it should be easier to spot the random part. In this example it were the values for title_de and title_en, which both contain a number from a factory_girl sequence.

Removing randomness

Some hints for removing any randomness that you detect:

• If randomness is caused by a factory_girl sequence, try to set a fixed value for that attribute in your test setup
• If randomness is caused by a timestamp, try to freeze the current time with libraries like Timecop or Sinon.js.
• If you absolutely can't get rid of the randomness, consider changing how VCR matches requests to recorded cassettes