Split your parallel tests by execution time and keep execution logs up to date

Both knapsack Show archive.org snapshot and parallel_tests Show archive.org snapshot have the option to split groups by historic execution time. The required logs for this might be outdated since you manually have to update and push them into your repository.

The following card includes an option how you can keep them consistently up to date with no extra effort locally and/or remotely.

How to always split by execution logs

Parallel Tests

The parallel_tests gem has the option flag --group-by to change the default splitting of test groups. If not specified other, the parallel execution will use the default option of this flag to split tests by file size if no execution logs are present, otherwise by execution logs.

For this purpose the gem allows you to record execution time when you call rspec with --format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log.
As the documentation on this Show archive.org snapshot states you can make this the default option by adding the following to your .rspec (or .rspec_parallel) file:

Copy--format progress
--format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log

Knapsack

For knapsack to generate reports you will have to set the variable KNAPSACK_GENERATE_REPORT to true like so:

CopyKNAPSACK_GENERATE_REPORT=true bundle exec rspec

You can set in your .spec_heper file (or within the features/support/.. folder for cucumber) ENV['KNAPSACK_GENERATE_REPORT'] = true to achieve this for every run.

Caveats and work arounds

There are some possible caveats with this practice. This section will take care on how to deal with these issues. Even if don't set the options above as a default this section includes some useful practical ways on how to correctly record the logs without side effects.

Using wrongly recorded data

The biggest caveat with this is, that you have to make sure to not commit wrongly logged data

Only commit complete testsuite runs:
You may only run specific specs on your local machine and may be running the complete testsuite mainly in the CI environment to wait until it's successfully finished.

Solution:
If this is the case you could configure the CI to push the correct files Show archive.org snapshot after it's finished.
You might want to consider doing this only after a couple of CI-pipelines to keep the number of these commits limited.

Don't record biased logs:
Also something else to consider here, is that you should prefer using mainly logs while your laptop is in steady performing state, i.e. don't run any other performance tackling programs while recording the logs.

Solution:

• An option might be during a short break.
• Also note that the CI may also have similar issues and might be good to choose a time, where there is less load on your CI.

Always included in diff

Another is that now they will always be included in your git diff for merge requests. There are two options how you can address this:

Option 1: exclude from diff

Luckily gitlab comes with a solution for this. You can ignore files for diffs within a project specific .gitattributes files like shown here Show archive.org snapshot :

Copytmp/parallel_runtime_rspec.log -diff
knapsack_rspec_report.json -diff

• Try changing the file and run git diff afterwards to check if everything worked
• You will have to merge them into you main branch at least one time to make it work on merge requests

Option 2: add to local git ignore file (only for parallel_tests)

This might be applicable if you work regularly on a project, such that it is sufficient to keep an updated local only version of logs or create that log file by running your testsuite as soon as possible when you start working on an issue.