Continuous Testing in dotCover and ReSharper Ultimate

Over the last couple of years, continuous testing has been the top anticipated dotCover feature. Now, the wait is finally over as dotCover 10 supports continuous testing in Visual Studio! Before we move on to the nitty-gritty details to show you how it works, let’s try to recall what continuous testing actually is.

Continuous testing

In a nutshell, continuous testing means testing ‘without interruption.’ It implies that tests are run in the background, so you have actual test results instantly, as soon as you’ve changed your code. This dramatically speeds up a developer’s workflow as there’s no need to manually rebuild the project and re-run all tests after making changes: the impacted tests are run in the background automatically (the testing session is triggered by some explicit action, e.g., saving changes).

If you practice test-driven development, the benefit is even more pronounced: the Red-Green-Refactor workflow becomes virtually seamless. All the boring routines go away, leaving you only with the fun part:

1. Write a test.
2. Write a stub.
3. Build.
4. Run the tests (they must fail).
5. Update the stub to return the expected result.
6. Build.
7. Run the tests (they must pass).
8. Write the code.
9. Build.
10. Run the tests (they must pass).
11. Refactor the code.
12. Build.
13. Run the tests (they must pass).

So, how to use continuous testing in dotCover? It’s very easy, with only three steps to follow.

1. Open Continuous Testing Session window

Continuous testing is managed via the Continuous Testing Session window. To open it, use the menu ReSharper | Unit Tests | Show Continuous Testing Session.

Continuous testing is off by default

2. Select continuous testing mode

By default, continuous testing is switched off. To make it work, select one of the modes in Mode:

Select continuous testing mode

A wide variety of combinations are available. For example, you can tell dotCover to automatically build and run dirty tests after you save the project. Or, run dirty tests only after you build the project.

What does ‘dirty test’ mean? It is the test that was impacted by code changes. For example, you have a test that covers some code, you change the code, and the test result is now obsolete, i.e. the test becomes ‘dirty.’ In addition, dotCover considers as ‘dirty’ all new tests, tests that were aborted by user (aborted), and tests that failed to complete for some reason (inconclusive).

3. Just work

Once continuous testing is enabled, you can simply start working. Initially dotCover will mark all tests as dirty.

The current tests status can be checked on the continuous testing icon in the status bar (bottom right corner of the main Visual Studio window). This handy assistant shows you whether there are any failed tests or all of them pass successfully. The Continuous Testing Session window displays the results in more detail:

Continuous testing status

As we already mentioned, your workflow with the enabled continuous testing is almost seamless. Suppose you change code that is covered by some tests and save the project (and you’ve selected the On ‘Save’ Build and Run Dirty Tests mode). dotCover builds the project and re-runs (only) dirty tests in the background:

Working with continuous testing

To navigate to a failed test, click the test in Continuous Testing Session window or click the number next to the continuous testing icon.

Failed test icon

Another option to navigate to the covering tests is to do it right from the code. Simply click on the code and select Show covering tests in the context menu. This opens the list of all tests that cover this line of code. Here you navigate to the corresponding tests or force them to run:

Show covering tests

Next steps

If you’re willing to try continuous testing right now, download and install ReSharper Ultimate.

IMPORTANT! If your tests that use external files fail during continuous testing (while being successful in a traditional unit testing session), you should properly set dotCover’s Shadow-Copy option. The following post describes in details how this option works.

Should you have any questions, please feel free to ask in the comments to this post. Your feedback is greatly appreciated!

This entry was posted in How-To's and tagged , , , . Bookmark the permalink.

25 Responses to Continuous Testing in dotCover and ReSharper Ultimate

  1. MichaelD! says:

    Really awesome! I was about to say that this article needs a video, but webinar is just as good. :) I have enrolled. This is a really awesome feature. Seems like what Visual Studio tried to do w/ its “Test on Build” which never seems to work. Once again ReSharper understanding MSFT’s technology better than MSFT itself.

    One suggestion: consider changing “dirty” to “modified.” Dirty is really an older term that is associated with the 90’s, and is not seen much of these days. Well, by me, at least. Personally, it makes me think something that is uncouth and/or needs to take a bath. 😛

    • Alexey Totin says:

      Thanks a lot for your involvement Michael! Actually, there was heated debate on how we should call these tests. The main objection against ‘modified’ (the other option was ‘affected’) was that these are not only tests modified by user or impacted by code changes but also aborted and inconclusive ones.

  2. Aleksandras says:

    I believe it works really nicely in your demos with a solution from 1-2 projects and with just 2 tests, but.. did you consider how it should work in a bigger solution? E.g. starting from 30 projects (still there are even bigger solutions) and with 2-3 Tests projects.

    For now it totally didn’t work for me (well, my machine doesn’t have 16+ CPU cores). Something like having “Solution-wide code analysis” enabled. I appreciate it’s just a early version of this feature, but I have some serious doubts it can ever work for bigger solutions..
    Any comments/objections on it? Would be glad to hear, that I’m wrong.:)

    • Jura Gorohovsky says:

      Aleksandras, as far as I know, QA tested continuous testing on a variety of solutions, with up to 50 projects.

      We perfectly accept that it might not work in all solutions, especially large solutions. If this is the case, then we’d be happy to see performance snapshots representing slowdowns related to continuous testing (can be made via ReSharper > Help > Profile Visual Studio). In cases where providing us solutions is possible (which is either if they’re available publicly or you’re willing to provide them under NDA), this would help as well. Based on what we can see in a performance snapshot, optimizations in terms of the way how dotCover’s continuous testing works might or might not be possible.

      In any case, as performance varies from solution to solution based on a myriad of factors, it makes sense to contact dotCover support to discuss what can be done to improve performance with continuous testing in each particular case.

      Thanks

      • Aleksandras says:

        Thanks, Jura, really appreciate your input.
        I believe, that our solution might have it’s own issues and should be optimized, but it just raised overall concerns about Resharper and Continous Testing – if it can work at all.
        I wasn’t aware there is a possibility to create profile snapshot for you so easily. Currently I disabled and uninstalled dotCover (since it was just trial and unfortunately it didn’t convince me to buy it..), but when I have some spare time I can try produce such snapshot for you. If it really could help you making Resharper+dotCover better – it sounds really promising.:)

        It would be also very nice to try it live on our real solution, but yes – it’s not open-sourced and there might be “NDA-issues”. Any other ideas how it can be tested? Maybe some screen-sharing sessions?

        By the way – a brief introduction of my company, just to ensure you that we are not just using your trials, but also quite power-users of Resharper. Currently I’m working in a project with ~30 developers (one of the reasons the solution had grown so big), everyone is with Resharper Ultimate license. And it’s just in this project, while we have more projects in our company.:)

        • Jura Gorohovsky says:

          Aleksandras,

          Yes, please, we would appreciate a snapshot whenever you can provide one. In the best case, it would allow us to see immediately what can be optimized. If it doesn’t, then we’ll see what additional info we might need.

          Screen sharing is nice but it would probably only allow us to see the end-user effects rather than know what’s causing them. However, if ReSharper and dotCover are run with verbose logging, then we might be able to analyze the logs and see any evident issues that they might expose. Anyway, the best first step is a performance snapshot, and if it doesn’t help, then a discussion with the support team should be the best guide what actions to follow up with.

          Thanks a lot for your willingness to cooperate. Other than not being currently able to run Continuous Testing, I hope you’re enjoying a smooth ride with ReSharper Ultimate on your team.

  3. Pingback: Dew Drop – November 20, 2015 (#2136) | Morning Dew

  4. Todd Aspeotis says:

    I hope one day ReSharper Ultimate might include something like https://comealive.io

  5. David Hewson says:

    We’re trying to use continuous testing but each continuous run has about 30-seconds to 1 minute of prep before the tests run and about 5 minutes afterwards which seems to be the coverage analysis. It spends most of that time writing a 500mb file into temp.
    Works fine on smaller projects though, just not this bigger one. To give you an idea of the size the total statements according to coverage is 689,939

    • Alexey Totin says:

      Hi, David
      This might be the result of how the caching is implemented in 10.0.0 and 10.0.1. If this is the case, 10.0.2 will solve the problem. If 10.0.2 (when it comes out) doesn’t help, we’ll really appreciate if you could take a performance snapshot when the slowdown takes place (can be made via ReSharper > Help > Profile Visual Studio) and send it to us.
      10.0.2 is currently available as EAP

  6. AnneTheAgile says:

    I registered for the webinar but couldn’t make it. Is it recorded? Is there a youtube or other channel of your webinars? Today’s on JS was very helpful. ty!

  7. Wes says:

    This is a sweet feature and is the reason that I want to upgrade to Ultimate.
    However, I’m having some problems w/ some tests that pass if I use the normal unit test session, but fail whenever in the Continuous Testing Session. I have 2 such tests failing out of 2400.

    I am noticing that the Continuous mode seems to copy the assemblies to a random location for execution of the test, however, the tests which file depend on files which are copied or created in sub folders relative to the test assembly are failing.

    For example, there is a test that says to create a db in “.\DatabaseTests\Someplace\Something.mdf” (this is using SMO)
    This test normally passes, but in my Continuous Test Session, it fails w/ this message:

    “Directory lookup for the file “C:\Users\Wes\AppData\Local\Temp\dodu3uyj.vqz\duxuys2y.eve\DatabaseTests\Something\Somewhere.mdf” failed with the operating system error 3(The system cannot find the path specified.).”

    Error code 3 is ERROR_PATH_NOT_FOUND

    Is there any way to control this behavior?
    What is dotCover doing here? I feel if I know what it was doing and I could update my tests to get around it.

    • Alexey Totin says:

      The problem is in how the default shadow copying works in Visual Studio. Long story short, 10.0.2 fixes the issue. Please note that it’s still in EAP https://confluence.jetbrains.com/display/ReSharper/ReSharper+10.0.2+EAP.
      If you want more details on how you can reference test data, please take a look at this comment in our tracker.

      • Wes says:

        I tried it but didn’t really get the result I wanted. I figured it’s easiest to remove these two tests from the scope.

        I reverted back to 10.0.1 (non-EAP) as my OCD was letting the exceptions submitter distract me.

        It seems that the continuous test session can get out of sync w/ regular unit tests. as an example, I made changes I expect to cause things to fail, but they report as passing in the continuous session, but will fail if I invoke it as a normal test, or vice versa. If I drop the session and create another it reports the right thing for a while.

        Sorta wondering why it does this.

  8. Pingback: First rule of programming - GilesCope.Ninja

  9. Evgeny says:

    Is it possible to make coverage highlighting feature like NCrunch (continuous instead of manual and coloured markers instead of coloured background)?

    • Jura Gorohovsky says:

      Currently there’s only colored background available as a way of highlighting; however, there’s a request for an alternative highlighting that you can vote up and/or comment: DCVR-7737.

  10. Jason Nappi says:

    I have a large solution with lots of tests. When I open the solution it automatically considers all tests as dirty. Can I instead consider then ‘green’ at start up and only turn dirty as I start making changes.

    • Alexey Totin says:

      Hi, Jason
      No, it’s not possible. For continuous testing, just data on whether tests pass or fail is not enough. dotCover also requires coverage data which it can get only during a coverage session. That’s why initially all tests are considered dirty.

  11. Alin says:

    Hello!

    I have Resharper Ultimate 2016.1.2.

    The .Net 4.6 solution has 90 projects, with 6400 MsUnit tests.

    Although, I’m very happy that netCover exists, the first main concern is speed execution of the Continous Testing Session. I’ve read this page and I see I’m not the only one.

    My machine is i7-3770 with 8 Gb of RAM, 256 Mb SSD.

    It takes tens of seconds to maybe 1 minute to execute and finnish 1 dirty test. I tasted NCrunch and the 2 are worlds apart. Still, I’d like to just use netCover Resharper Ultimate…

    I’m not sure how to give a profiling snapshot. Please guide me if possible.

    Thanks!

  12. Vijay Venkatesh Prasad N says:

    Hello!
    I have Resharper Ultimate 2016.1.2.

    I am trying to run the continuous testing session but it is failing because its not referring the Unit test sessions test settings.

    Could you please let me know how to update the test setting for continuous testing session?

  13. Pingback: Visual Studio Live Unit Testing | Kultainen koodi

Leave a Reply

Your email address will not be published. Required fields are marked *