Running Code Coverage from the Console with dotCover

As of the beta* of dotCover, we included a Console runner to run coverage using the command line, allowing for instance, setup of dotCover in a Continuous Integration environment. Let’s see how it works.

*To get all these features in this post, you need to download the latest nightly build

Console Runner

The Console runner is located under the installation folder (%Program Files%JetBrainsdotCoverv1.0bindotCover.exe). The best option is to add it to the system path so as to be able to run it from anywhere. The runner accepts several commands based on the operation we want to perform. Each of these commands in turn takes one parameter, which is an XML configuration file. The commands are:

  • cover: Coverage of specified application
  • merge: Merges snapshots
  • report: Creates an XML report of the coverage results
  • list: Lists snapshots
  • delete: Deletes snapshots
  • analyse: Provides an all-in-one analysis and output

We can also obtain a list of commands by typing dotCover.exe help on the command line:

image

As shown in the figure above, we can find out more about each command by typing help followed by the command. All commands (including help) have corresponding shortcuts. For instance, to get information about analyse, we can type:

dotCover.exe help analyse

or

dotCover.exe h a

Since all commands take as parameter an XML file, when requesting help, all they do is generate a sample XML with comments indicating what each element means. The previous command would therefore output:

image

The obvious advantage to this is that we can easily get a new configuration file setup by just asking for help and piping the output to some file. In fact, adding a third filename parameter, dotCover will do this for us, without us having to touch up the output (remove headers, etc.).

Coverage in one command

The dotCover console runner is flexible in that it complies with most common requirements in a Continuous Integration setup. Normally in projects, we have multiple test projects we need to cover. We might then want to see the results separately or merge them as one report. dotCover allows for this kind of flexibility. However, in many cases, all we want to do is run coverage on a single test project and see the results. We’ll see how to do that in a single step in this blog post and in a future post we’ll get into the details or merging, listing, etc.

Let’s see how to do this step by step. I’ll be using a sample app that I’ve prepared for this blog post (using MSpec). You can adjust the parameters to your own project paths and unit testing framework as required.

  1. Type dotcover help analyse coverage.xml on the command line. I’m doing this in the root folder of my solution.

    image

  2. Add coverage.xml to the Solution and open it up in Visual Studio
  3. Eliminate all elements except the following

     image

  4. Fill in the elements with the corresponding values:

    Executable: Is the path to the unit testing framework runner (in this case mspec.exe)
    Arguments: Arguments passed to the runner, the test assemblies (specs.dll)
    WorkingDir: Optional path to the working folder (provide this to not have to fully qualify test assemblies)
    Output: XML Report containing coverage information. Relative to the coverage.xml path (output.xml)
     image

  5. That’s it in terms of configuration. Now all we need to do is run it by typing dotCover analyse coverage.xml. If all is configured correctly, we should see the following output:

    image

    and there should be an output.xml file located in the same folder. Opening it up we can see the coverage results

    image

 

Filtering from the Console

Looking at the results, we can see that there is a lot of noise, that is, assemblies that we’re not interested in such as the tests, MSpec assemblies, etc. In a previous blog post, we saw how to filter these out using dotCover GUI. Let’s see how we can do this via the command line. If we generate the default coverage configuration, we can see that there is a filter section:

image

These are used to filter assemblies. We can indicate which assemblies we want covered and which we don’t. In order to filter out all assemblies and concentrate only on our actual code (ClassLib), we can do one of two things:

  1. Filter out all non-required assemblies individually by adding them to the ExcludeFilters section, as in:
     image
  2. Exclude everything that’s not specifically in the Include section. This is done by leaving an ExcludeFilter section blank and placing what we want covered in the IncludeFilter:

    image

Based on the needs and depending on which of the two lists (Include or Exclude) are smaller, we can opt for option 1 or 2. The ouput should now be filtered:

image

 

Giving it some style

Only thing left to do is to format the output so that it can be viewed nicely inside a browser. This can be done easily using XSLT and some appropriate design skills (something I lack).

image 

Included in the project you have an XSLT to create some HTML as well as one to covert the XML to JSON. Having JSON output allows us to combine it with something like jQuery to display a nice treeview or grid with the results.

This entry was posted in dotCover Tips&Tricks, How-To's and tagged , , . Bookmark the permalink.

27 Responses to Running Code Coverage from the Console with dotCover

  1. Joe White says:

    Very cool. I’m not clear on the difference between the “cover” and “analyse” parameters, but I hope you’ll cover that in a future post. And I’m glad to hear you’ve got a “merge” parameter. I hope it can handle multiple runs against the same project, not just different projects.

    One question:

    Does Executable really have to be a fully-qualified path? If so, that’s a pain — our checkout paths aren’t even the same on all our dev boxes, and that’s even if we’re working with the latest code and not a branch. And the CI machines have still different paths. I guess we could always write a Ruby front-end to generate the config file every time we run, but still, requiring a full path seems like a very strange choice.

  2. hhariri says:

    @Joe,

    Yep will cover that in a second post. Normally the procedure is to do cover, merge, report. However that’s for scenarios where you have multiple test projects etc. If in 80% of the cases all you want is to just run coverage on a project in one step, that’s where analyse comes in. In fact we added it to prevent you have to issue 3 separate commands. It’s like a shortcut. But as I said, I’ll cover these in more details in a second post.

    Regarding the executable, it would need to be in the folder where you run this from or full path. In your case, you’d normally have a setup on the CI server or as you say, copy it over.

  3. Ruslan Isakiev says:

    Hello Joe,

    “Cover” command performs coverage analysis and creates snapshot(s), the result is written to the “Output” parameter. You can use this result as an input for “Merge”, “Report”, “List” and “Delete” commands.
    “Analyse” is all-in-one command which includes performing coverage analysis, merging snapshots (in case if several snapshots were produced during analysis (one snapshot per process)), XML report generation and cleanup. “Output” parameter in this case is an XML report file name.

  4. Lokesh says:

    hi,

    Following is the Log seen (end of it) when i run the Dotcover from commandline

    Disposing the test runner.
    Stop time: 12:15 PM (Total execution time: 491.033 seconds)

    654 run, 646 passed, 8 failed, 0 inconclusive, 0 skipped

    Analysed application exited with code ’1′
    Coverage session finished [3/23/2011 12:15:19 PM]
    Snapshot merging started [3/23/2011 12:15:19 PM]
    Source snapshots number: 2
    Merging snapshots 1-2
    Snapshot merging failed [3/23/2011 12:15:19 PM]. Null AssemblyName

    So i dont get the XML report
    What may be the problem??

  5. Hadi Hariri says:

    @Lokesh,

    What does your dotcover configuration look like? What unit test runner are you using?

  6. Eladio says:

    On the CI, how does dotcover install and run?

    Do I need admin rights to install and configure the licenese?

    I use Hudson for my build server so I need to deploy dotcover on the fly (some sort of xcopy), get it configured and executing on machines that sit on a farm.

    Is this possible?

    Eladio

  7. Hadi Hariri says:

    @Eladio,

    TeamCity includes dotCover already. If you’re using something different (Hudson) in your case, unfortunately right now the only way to do it is installing it which requires Visual Studio. We’re working on removing this requirement.

  8. Ash says:

    How do I run tests with coverage from the command line for a silverlight app?

  9. Gerhard says:

    Is it still a requirement to install dotCover on our BuildMachine ? This would be really show stopper … We are also using Hudson.

    Thx for a short answer.

  10. Gerhard says:

    Sorry. I wanted to say – to install VS2010 on our build machine…

  11. Hadi Hariri says:

    You need to install @dotCover on the build machine to run it still, yes.

  12. DotCoverEvaluator says:

    Hi,
    I am trying to set up DotCover to run it from outside VS.
    I am using nunit as unit test framework.

    I followed the steps explained in this page, but I get an error message (There is no Application Shell running.) when I run dotCover analyse coverage.xml

    ———-
    $ dotCover analyse coverage.xml
    JetBrains dotCover Console Runner v1.2.384.4. Copyright (c) 2009-2012 JetBrains s.r.o. All rights reserved.

    Unhandled exception: There is no Application Shell running.
    ———–
    What have I missed?
    I have a trailed version of dotCover now.

    Best regards!

  13. DotCoverEvaluator says:

    Hi,
    I managed to run it now, but have another problem
    But still some error, I get the following message

    .
    .
    Analysed application exited with code ‘-1′

    and when I look at the generated html the coverage is 0% while if I run it from VS I have 5%.
    What does code ‘-1′ means ?

    Thanks!

  14. lukas says:

    Hey,

    is it possible to have a full file path in the xml report? Ex. File foo/boo/moo.cs covered in 60%, so not the class and namespace but a file path ?

  15. srikara says:

    Is it still a requirement with dotCover 2 to install Visual Studio in the server to get code coverage through console? Or is there a way I can get code coverage with out installing Visual Studio?

  16. Subith says:

    Hi,

    I have installed dotcover 2.2 in VS2012. Ad created a coverage.xml as follows

    C:Program Files (x86)JetBrainsdotCoverv2.2BindotCover.exe
    E:SVNReposcscr-toolstrunksrcvstudioBOMSystemTestBOM-TestbinDebugBOM_Test.dll
    E:SVNReposcscr-toolstrunksrcvstudioBOMSystemTestBOM-TestbinDebug
    out.xml

    I ran the following commandline
    “C:Program Files (x86)JetBrainsdotCoverv2.2BindotCover.exe” analyse E:SVNReposcscr-toolstrunksrcvstudioBOMcoverage.xml

    And i am getting thee following error

    Unknown command: E:SVNReposcscr-toolstrunksrcvstudioBOMSystemTestBOM-TestbinDebugBOM_Test.dll
    Type ‘dotCover help’ for usage.

    BOM_Test.dll is my unitest library . Can anyone please hep me to fix this issue?

    Thanks,
    Subitha

  17. Gopinath says:

    I have installed DotCovere 1.2 version on my machine and it’s also showing in VS 2010. when i ran the tests, the tests passed successfully but i was not able to find Dot Cover coming into scene anywhere . Do i need to configure it properly in VS 2010 . how can i do that without using Re-sharper .

  18. Gopinath says:

    How can i generate cover reports in VS 2010 using Dot cover . I have the project and the tests ready .

  19. Lex says:

    I am evaluating dotcover for use with Windows 8 store Apps solution.
    The solution supports a framework, that supports multiple windows 8 store apps.

    I attempted a command line run but had no success using vstest.console.exe as the binary.
    coverage.xml

    vstest.console.exe
    C:artifactsUnitTest_BaseCommunication.Tests_25.0.0.0_AnyCPU_TestBaseCommunication.Tests_25.0.0.0_AnyCPU.appx /InIsolation
    C:artifactsUnitTest_

    Is this possible? Its really unclear to me with current level of documentation.

  20. Prakash Moharer says:

    Hi Guys, I need a bit of assistance. The issue that I am facing is while running Individual Unit/Integration Test Cases it is passing, but it failed while it is run using dotCover. Please advise

  21. Hi Prakash,

    I will ping you via e-mail.

    Best regards
    Maarten

  22. KZSun says:

    If for some reason we are using both NUnit Framework and Visual Studio MS Test Framework in the same test project. Then how should I add both Executables?

    NUnit Framework Exe

    MSTestExe

    Thx in advance

  23. nuno says:

    Is there any way to run the console app without launching the UI? I have dotcover installed on our build machine, and i need a console app that runs silently and deposits the xml output for Jenkins to parse. Initially, i’d like to make the report available to devs from the Jenkins console, but eventually, i’d like to break the build if the coverage dipped below a certain threshhold percentage. Is any of this possible?

  24. Hi Nuno,

    You can use the approach outlined in this blog post to run dotCover on the console (it should not show the UI). The tool can output HTML reports or XML reports, depending on hwo you want them to surface in Jenkins. More information can be found in https://www.jetbrains.com/dotcover/webhelp/dotCover__Server_Test_Coverage.html

    If you want the build to fail whenever a coverage threshold isn’t met, you will currently have to parse the XML report for that data. Do vote for this feature request http://youtrack.jetbrains.com/issue/DCVR-6223 to make this an easier task.

    Feel free to send me an e-mail if you need assistance with this.

    Best regards,
    Maarten

  25. Hi,
    I would like to specify the code coverage parameters for a web application in an XML file to be used with the analyse command in the command console. Could anyone help me on the same.

    Thanks

    Patric

  26. Vladimir says:

    What should i do to take coverage of my webapp (IIS)?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">