dotCover: Code Coverage for .NET

Last week we announced a new product by JetBrains for .NET developers: dotCover, a code coverage tool. We are aiming to open the EAP this week and some of our Academy Members have already been playing with it since last week. In this post I’m going to provide some more info on how it works.

Code Coverage 101

Code Coverage is a means of measuring what percentage of your code is covered by tests. Technically,the higher the percentage, the less likely you are to find hidden bugs when it’s too late. You could therefore assume that 100% coverage is ideal, and you’d most likely be wrong. Having your entire code base covered by tests doesn’t necessarily mean that you are safe. Code Coverage only analyzes which parts of your code are covered by tests; it doesn’t tell you whether the code you’ve written is correct. Additionally, having a higher percentage often gives a false sense of security, that the code is now bug-free and doesn’t require any other type of testing.

Does this mean, therefore, that Code Coverage is useless? Quite the contrary. Code Coverage tools normally provide, at a minimum, two pieces of relevant information: the percentage of code covered and the actual lines of code covered. However, what is important is not so much the code that is covered, but what is NOT covered. If I have a code base with 70% code coverage, what is more valuable to me than knowing that 30% of my code is not tested, is finding out what code that is. This allows me to find potential issues that would otherwise go unnoticed.

Testing code for coverage with dotCover

Disclaimer: At the time of writing this blog post, dotCover is still pre EAP so some things may not work as expected

It should somehow be apparent that coverage tools need to work in conjunction with testing frameworks and unit tests. I’m going to base this post on the sample MVC application that ships with ASP.NET MVC 2, the one that comes with some default controllers, not the blank one, you know, the one that doesn’t even bother asking us whether we want a unit test project or not. The project ships with two controllers, HomeController and AccountController along with some corresponding tests.

Similar to dotTrace, dotCover works nicely with ReSharper. When you install dotCover, you see several options appear in the ReSharper menu options as well as the Test Runner window. First of all, under ReSharper | Unit Tests menu option, there is a new option to run coverage reports, either for specific tests or for the entire solution.

Similarly, the Test Runner Window offers us the same options. We get a new icon to run tests under coverage, as well as a new tab that will display the output of the coverage report.

Based on the information we want to obtain, we can run coverage reports on the entire solution or only certain tests. Running on the entire solution will give us a measurement of how much of our entire code base is covered. In order to do so, we can select Cover All Tests from Solution. You can assign whatever keyboard shortcut you want to this action by selecting the ReSharper_UnitTest_CoverSolution command, as shown below:

Executing the command will run the unit tests and once completed will then proceed to run a coverage report.

We can also run coverage on individual tests or classes, by selecting them and choosing to Cover with dotCover

Let’s try this on the AccountController tests. When we run the coverage, the unit tests will execute inside the ReSharper test runner and once completed, it will then proceed to run coverage reports. To view the results, we can click on the dotCover tab:

The window groups coverage reports based on the option selected in the drop-down list. We’re interested in the amount of coverage of our main assembly, not that of the tests (in the future we’ll be able to filter test assemblies out). By expanding the node that references our assembly, we get more insight into the actual coverage. Notice that HomeController is at 0% coverage. This doesn’t mean that there are no tests, but that in this specific test run, none of the tests have been executed. This is because we chose to run only the AccountController tests.

dotCover provides us with two measurements: percentage of code covered and number of statements. As such, if we have 83 statements in AccountController out of which 54 are covered with tests, that should indicate 65% coverage. Let’s dig a little deeper to see what is not covered.

We can see that out of the eight actions, four have complete coverage and two have partial coverage. One of these with partial coverage is ChangePassword. There are two overloaded actions for it and it’s a common pattern in ASP.NET MVC applications; one corresponding to a GET request and another to a POST. The second one in the coverage report corresponds to the latter, and shows only 53% coverage. To see in more detail why this is, we can select the Highlight Code icon on the test runner panel:

This will highlight the paths of the code that are covered/not covered by tests. Next step is to select the ChangePassword action by double-clicking on it

The lines colored in Cyan mean that they are covered by tests, whereas those in Lilac are not in the execution path of any of the executed tests. In the case of ChangePassword we can see that the test only covers the situation where the model state is valid. This allows us to easily identify non-happy paths of our application and make sure that when relevant, the correct tests are in place for it.

What’s coming…

dotCover is still in its infancy and we’ve got a lot planned in terms of features and functionality. Quite a few people have asked whether it will support running coverage reports from a console, which is important for continuous integration. dotCover will support running from a console with some basic functionality, and we have plans to include specific features for CI processes.

If you aren’t yet following the dotCover Twitter account, make sure you do as we’ll release information and updates as things move forward.

This entry was posted in dotCover Tips&Tricks, News and Events and tagged , . Bookmark the permalink.

25 Responses to dotCover: Code Coverage for .NET

  1. Omer Mor says:

    I suggest you include branch coverage along with the line coverage that you provide.
    We are often required for branch coverage by our customers.
    This feature is already available for .Net by the NCover tool.

  2. @Omer

    Thank you for the suggestion!
    Could you please request the feature using dotCover issue tracker? Doing so will dramatically improve the chances this feature will be implemented.

  3. It is nice to see another option in the “coverage” space, however what I would find useful is a list of the compelling reasons to consider dotTrace over the the incumbant and the coverage tools that come with VS.

    What differentiates dotTrace from the other coverage tools?

    -dave

  4. hhariri says:

    @Omer

    Thanks for the feedback. Branch coverage is in fact on the roadmap to be supported.

  5. hhariri says:

    @David,

    Thanks for your interest. I’m guessing you mean dotCover, since dotTrace is for profiling. What does it offer against VST? Currently it works with all unit testing frameworks that ReSharper supports, that means MSTest, NUnit, xUnit, to name a few, whereas, and unless I’m mistaken VS coverage only works with MSTest currently.

  6. Grant Palin says:

    This is interesting news, but it makes sense that JetBrains would develop a code coverage tool to complement the others in the stable.

    Will dotCover work just as a Visual Studio plugin, or will it by any chance be source-control friendly? I like to keep my build-related tools in source control alongside my code.

  7. hhariri says:

    @Grant,

    Yes it works as a VS plug-in. How do you mean source-control friendly? You can always check-in the MSI if you like….

  8. Grant Palin says:

    I use some other tools, such as NAnt and NUnit, which I check into source control along with my code. So with a single checkout I can compile the code and run unit tests, without any other dependencies. Is there any provision for dotCover to be used in a standalone fashion, or is it intended to be used in Visual Studio?

  9. hhariri says:

    @Grant,

    Ah you mean command line friendly? Yes. There’s a console runner and we’ll also provide integration with TeamCity

  10. Grant Palin says:

    We’re getting closer. The console runner is good to know about, but what I am principally concerned about is…can the tool be used without installing it? Can it be used standalone? This can be done with NAnt and NUnit, among others, and I prefer to keep non-compiler tools close to my code, so I wonder if the same can be done with dotCover.

  11. hhariri says:

    @Grant,

    It’s an MSI. There’s no zip you can unpack and run. If you can unpack the MSI (which is possible), then you could run it most likely, since as I mentioned there’s a console runner and as such works w/o the need for Visual Studio

  12. Grant Palin says:

    Thanks for the clarification, that’s essentially what I was looking for. Sounds like I’ll be doing some tinkering with it when I get my hands on it!

  13. Adam says:

    In regards to the console runner, have you considered packaging in its own zip package (Seperate from the VS plugin) so that it can be easily deployed without installation? This would enable easy deployment in CI environments. I believe that providing two packages (console only zip & vs plugin msi) would make things very simple and enable usage in all scenarios.

  14. hhariri says:

    @Adam,

    I’m not sure if this is on the initial list of features, but I’ve taken note however.

    Thanks.

  15. Sean Feldman says:

    For CI VS.NET and MSI are not relevant. I agree with stated above, being able to “unzip and use” will allow anyone who’s practising CI to use dotTrace. Command line runner, tasks for NAnt/MsBuild and no-installer would be a great feature to have.

  16. Seems very interesting, Hope the integration with TeamCity make dotCover as outstanding as another tools for JetBrains.

    One question: dotCover will support Test written in other languages than C# y VB, like F#?

  17. Hadi Hariri says:

    @Horacio

    Checkout the other posts about the console runner that allows TC integration for now. We’re working on tighter integration too as mentioned

  18. Kenneth LeFebvre says:

    Will dotCover support SmartDevice projects?

    That feature, alone, would get you many customers who have been left out by every other coverage tool on the market!

  19. Hadi Hariri says:

    @Kenneth

    Yes, it does cover SmartDevice Projects.

  20. Hein says:

    If using dotCover in the standalone console mode, is there any way of viewing the snapshots to see the actual source code lines covered, or can you only produce xml summaries of the high-level statistics?

  21. Hadi Hariri says:

    @Hein,

    It should contain all relevant information.

  22. Giles says:

    Thanks for this, …. I just tried dotCover locally on some of our tests, and it works surprisingly well for F# code (and our total coverage isn’t too much of a disaster either, although the F# compiler generates a lot of code for us, such as Equals/GetHashCode, that we never need to call).

    – Great news – but potentially could we tidy up for F# a tad?

  23. Vaibhav says:

    Can dotCover be used for WinRT components written in C# for Win8 Apps? If yes, what are the steps when only dotCover is used without ‘ReSharper’ ?

  24. @Vaibhav,
    Yes, dotCover can perform coverage analysis for the WinRT applications.
    Presence of ReSharper is not required for this feature.
    Configuration dialog is available from the VS main menu “dotCover->Cover Application->Windows Store Application”.
    In case if you have set of communicating WinRT applications, there is an option to analyse any .net process.
    In order to get coverage analysis for the Metro-tests you can try dotCover 2.2 EAP:
    http://confluence.jetbrains.net/display/DCVR/dotCover+Early+Access+Program
    Please let us know if you’ll have any questions.

  25. Sanjay Kumar Yadava says:

    Hi,
    I want to generate the Test code coverage report for Manual test cases without closing the Test Application. Currently I start my test application using the dot cover console runner and test user performs the actions on the test application. Finally user closes the test application and dot cover generates the xml code coverage file at the specified path for these actions.
    But I want that at intermediate time or point of the action on test application, if we can notify to dot cover that one test case is completed, and coverage file should be generated for those steps. Now when User again starts other actions of next test case and again dot cover should start finding the coverage of methods for second test case from here (it should not keep the coverage of previous test case steps) and again when user notify, coverage file should be generated for second test case so on..
    So purpose is that user has to not close the application and again start application for the next test case verification, user can execute and verify the test cases steps in one go and he need not to start application again and again. As dot cover has integration with Visual studio, Get Snapshot, Drop Snapshot, Start Profiling etc. Please help me how can I achieve this functionality using the dot cover tool runner console.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">