Improved continuous testing performance in dotCover 2017.2

Continuous testing (CT) is a super-helpful feature but only under one condition: sufficient performance. Unfortunately, in dotCover 2017.1 and earlier, CT could be a real pain on large and very large solutions (30+ projects). The problem lies in three performance-critical phases of CT (actually, this is true not only for CT but for all tests run under coverage), which are:

  1. Coverage process startup.
  2. Test execution (could be much slower than usual due to the additional statement-level instrumentation).
  3. Post-processing of coverage snapshots.

In 2017.1 and earlier, (3) was the worst performing phrase because multiple coverage processes running in the background created as many coverage snapshots. Even though coverage time wasn’t too big, the merging of subsequent snapshots was very slow due to the snapshot format. On real solutions, post-processing could take up to several minutes (and prevented running tests during that time).

Our support team’s recommendation in all such cases was to limit the number of coverage processes. This could be done by setting Run up to N assemblies in parallel to 1 in ReSharper’s unit-testing options. Limiting the number of snapshots to just one would prevent snapshots from merging.

dotCover. Run N assemblies in parallel

dotCover 2017.2 has finally addressed the issue,* changing its snapshot format to reduce merging time. Reports from users who have tried the updated feature indicate that the coverage phase during continuous testing has decreased from several minutes to under 30 seconds.

*The slowdowns during phases 1 and 2 will be subject to optimization in future dotCover releases.

That’s why now, on the contrary, we can recommend that you set Run up to N assemblies in parallel so that it matches the number of logical processors of your computer. Of course, each user case is unique and on some solutions (e.g., if unit tests from different assemblies perform intensive disk operations) the performance increase will not be as noticeable. Nevertheless, in most cases, this setting should positively affect the overall performance of continuous testing.

Logical processors

To try accelerated continuous testing, download and install ReSharper Ultimate 2017.2. Should you have any questions, please feel free to ask in the comments to this post. Your feedback is greatly appreciated!

Comments below can no longer be edited.

9 Responses to Improved continuous testing performance in dotCover 2017.2

  1. Avatar

    George Cook says:

    September 13, 2017

    Is it possible to change the default generate code behavior for fields? they are tucked away under the other section, it drives me crazy – the first option is generate var, which is 3 keystrokes and something I’d never ever want.
    generate field, which I want 90% of the time, and would be considerable more work to do than type the word var (I’d have to scroll up, lose my cursor, type the field delcaration, and return to where I was) – to do that I have to press alt enter start typing other, then type field

    I’m not raising a bug, as I assume that there must be a way to do this in the IDE that I don’t know.

  2. Avatar

    Paco says:

    September 15, 2017

    Very glad to hear about it! Hope to see this and future improvements included also in dotCover’s version for TeamCity.

    By the way, in MS Windows you have the environment variable NUMBER_OF_PROCESSORS to get the number of logical processors of your computer.

  3. Avatar

    Pavel says:

    September 18, 2017

    Running dotCover freeses UI

  4. Avatar

    Justin says:

    February 22, 2019

    How can this setting be applied in TeamCity as our coverage report generation with dotCover takes 7 minutes.

    • Alexey Totin

      Alexey Totin says:

      February 22, 2019

      Hi Justin,
      no, there’s no such option in TeamCity. If coverage report generation requires more time than usual, then it’s probably a bug or some misconfiguration. Could you please contact dotCover support?

  5. Avatar

    justingol says:

    February 25, 2019

    Im a bit confused. This performance fix addresses what seems to be the same issue I have which is report generation takes 7 min but with the fix listed here it goes down to 30 sec. I was hoping this is an argument we could pass in or improvement that could be made to leverage this in TeamCity.

    • Alexey Totin

      Alexey Totin says:

      February 25, 2019

      Hi Justin,
      the majority of unit testing frameworks, i.e. their unit test runners are able to run tests in parallel. In the ‘continuous testing’ mode, dotCover disables this feature (in order not to break per-test info) but uses the ReSharper’s ability to ‘Run up to N assemblies in parallel’ instead. TeamCity runs coverage analysis in a ‘standard way’, so the option is not needed there. Nevertheless, it is possible that the cause of your performance problems is that the NUnit runner itself is not excluded from coverage analysis. To say for sure, we need to take a look at TeamCity logs. If it’s possible, could you please contact our support team and send the logs to them?

      • Avatar

        Justin says:

        January 17, 2020

        I thought the coverage generation on the solution could be done in parallel through the command line but its ok as I can do this another way.

        I see what you mean that the merge takes much less time and it looks like thats the case for me too. Has the new format also improved the report generation as that has improved for me too. My timings are 4m21s to generate the coverage reports (no building or running tests), 4s to do the merge and 19s to generate the report.

Discover more