Cucumber CI job quirks

Most of our CI pipelines don't use the --retry flag for Cucumber and instead build their own retry via the tmp/failing_features.txt file.

Benefits:

It's possible to only use -f pretty for the rerun.

Drawbacks:

MAJOR: With our current setup, when the main run fails without writing a tmp/failing_features.txt (e.g. due to a syntax error), the CI job will pass
MINOR: With our current setup, we lose the test coverage of the main run

A fix for the passing CI despite syntax error could look like this:

cucumber:
  # ...
  script:
    - main_cucumber_run_passed=true
    - bundle exec rake "knapsack:cucumber[--strict --tags 'not @real-stripe-requests' -f progress -f rerun -o tmp/failing_features.txt]" || main_cucumber_run_passed=false
    - 'if [[ $main_cucumber_run_passed =~ false && ! -s tmp/failing_features.txt ]]; then (echo -e "\033[0;31mCucumber failed, but produced no non-empty tmp/failing_features.txt file. You probably have a syntax error in one of your features.\033[0m"; exit 1); fi'
    - 'if [[ $main_cucumber_run_passed =~ false ]]; then (echo -e "\033[0;33mRerunning failed tests:\033[0m"; bundle exec cucumber @tmp/failing_features.txt --strict --tags "not @real-stripe-requests" -f pretty -f junit -o tmp/artifacts/reports); fi'

Alternatives to consider:

Use Cucumber's native --retry flag. You'll lose the -f pretty on the rerun, but we'll lose that in the future anyways when migrating to RSpec

Klaus Weidinger

Say thanks

Last edit

2025-12-05

Klaus Weidinger

License

Source code in this card is licensed under the MIT License.

Posted by Klaus Weidinger to makandra dev (2025-12-05 14:46)