Running Code Coverage from the Console with dotCover

Hadi Hariri July 28, 2010

As of the beta* of dotCover, we included a Console runner to run coverage using the command line, allowing for instance, setup of dotCover in a Continuous Integration environment. Let’s see how it works.

*To get all these features in this post, you need to download the latest nightly build

Console Runner

The Console runner is located under the installation folder (%Program Files%JetBrainsdotCoverv1.0bindotCover.exe). The best option is to add it to the system path so as to be able to run it from anywhere. The runner accepts several commands based on the operation we want to perform. Each of these commands in turn takes one parameter, which is an XML configuration file. The commands are:

cover: Coverage of specified application
merge: Merges snapshots
report: Creates an XML report of the coverage results
list: Lists snapshots
delete: Deletes snapshots
analyse: Provides an all-in-one analysis and output

We can also obtain a list of commands by typing dotCover.exe help on the command line:

As shown in the figure above, we can find out more about each command by typing help followed by the command. All commands (including help) have corresponding shortcuts. For instance, to get information about analyse, we can type:

dotCover.exe help analyse

dotCover.exe h a

Since all commands take as parameter an XML file, when requesting help, all they do is generate a sample XML with comments indicating what each element means. The previous command would therefore output:

The obvious advantage to this is that we can easily get a new configuration file setup by just asking for help and piping the output to some file. In fact, adding a third filename parameter, dotCover will do this for us, without us having to touch up the output (remove headers, etc.).

Coverage in one command

The dotCover console runner is flexible in that it complies with most common requirements in a Continuous Integration setup. Normally in projects, we have multiple test projects we need to cover. We might then want to see the results separately or merge them as one report. dotCover allows for this kind of flexibility. However, in many cases, all we want to do is run coverage on a single test project and see the results. We’ll see how to do that in a single step in this blog post and in a future post we’ll get into the details or merging, listing, etc.

Let’s see how to do this step by step. I’ll be using a sample app that I’ve prepared for this blog post (using MSpec). You can adjust the parameters to your own project paths and unit testing framework as required.

Type dotcover help analyse coverage.xml on the command line. I’m doing this in the root folder of my solution.
Add coverage.xml to the Solution and open it up in Visual Studio
Eliminate all elements except the following
Fill in the elements with the corresponding values:
Executable: Is the path to the unit testing framework runner (in this case mspec.exe)
Arguments: Arguments passed to the runner, the test assemblies (specs.dll)
WorkingDir: Optional path to the working folder (provide this to not have to fully qualify test assemblies)
Output: XML Report containing coverage information. Relative to the coverage.xml path (output.xml)
That’s it in terms of configuration. Now all we need to do is run it by typing dotCover analyse coverage.xml. If all is configured correctly, we should see the following output:

and there should be an output.xml file located in the same folder. Opening it up we can see the coverage results

Filtering from the Console

Looking at the results, we can see that there is a lot of noise, that is, assemblies that we’re not interested in such as the tests, MSpec assemblies, etc. In a previous blog post, we saw how to filter these out using dotCover GUI. Let’s see how we can do this via the command line. If we generate the default coverage configuration, we can see that there is a filter section:

These are used to filter assemblies. We can indicate which assemblies we want covered and which we don’t. In order to filter out all assemblies and concentrate only on our actual code (ClassLib), we can do one of two things:

Filter out all non-required assemblies individually by adding them to the ExcludeFilters section, as in:
Exclude everything that’s not specifically in the Include section. This is done by leaving an ExcludeFilter section blank and placing what we want covered in the IncludeFilter:

Based on the needs and depending on which of the two lists (Include or Exclude) are smaller, we can opt for option 1 or 2. The ouput should now be filtered:

Giving it some style

Only thing left to do is to format the output so that it can be viewed nicely inside a browser. This can be done easily using XSLT and some appropriate design skills (something I lack).

Included in the project you have an XSLT to create some HTML as well as one to covert the XML to JSON. Having JSON output allows us to combine it with something like jQuery to display a nice treeview or grid with the results.

#Code Coverage #dotCover #unit testing

Comments below can no longer be edited.

32 Responses to Running Code Coverage from the Console with dotCover

Joe White says:
July 28, 2010
Very cool. I’m not clear on the difference between the “cover” and “analyse” parameters, but I hope you’ll cover that in a future post. And I’m glad to hear you’ve got a “merge” parameter. I hope it can handle multiple runs against the same project, not just different projects.

One question:

Does Executable really have to be a fully-qualified path? If so, that’s a pain — our checkout paths aren’t even the same on all our dev boxes, and that’s even if we’re working with the latest code and not a branch. And the CI machines have still different paths. I guess we could always write a Ruby front-end to generate the config file every time we run, but still, requiring a full path seems like a very strange choice.
hhariri says:
July 28, 2010
@Joe,

Yep will cover that in a second post. Normally the procedure is to do cover, merge, report. However that’s for scenarios where you have multiple test projects etc. If in 80% of the cases all you want is to just run coverage on a project in one step, that’s where analyse comes in. In fact we added it to prevent you have to issue 3 separate commands. It’s like a shortcut. But as I said, I’ll cover these in more details in a second post.

Regarding the executable, it would need to be in the folder where you run this from or full path. In your case, you’d normally have a setup on the CI server or as you say, copy it over.
Ruslan Isakiev says:
July 28, 2010
Hello Joe,

“Cover” command performs coverage analysis and creates snapshot(s), the result is written to the “Output” parameter. You can use this result as an input for “Merge”, “Report”, “List” and “Delete” commands.
“Analyse” is all-in-one command which includes performing coverage analysis, merging snapshots (in case if several snapshots were produced during analysis (one snapshot per process)), XML report generation and cleanup. “Output” parameter in this case is an XML report file name.
Lokesh says:
March 23, 2011
hi,

Following is the Log seen (end of it) when i run the Dotcover from commandline

Disposing the test runner.
Stop time: 12:15 PM (Total execution time: 491.033 seconds)

654 run, 646 passed, 8 failed, 0 inconclusive, 0 skipped

Analysed application exited with code ‘1’
Coverage session finished [3/23/2011 12:15:19 PM]
Snapshot merging started [3/23/2011 12:15:19 PM]
Source snapshots number: 2
Merging snapshots 1-2
Snapshot merging failed [3/23/2011 12:15:19 PM]. Null AssemblyName

So i dont get the XML report
What may be the problem??
Hadi Hariri says:
March 24, 2011
@Lokesh,

What does your dotcover configuration look like? What unit test runner are you using?
Eladio says:
June 3, 2011
On the CI, how does dotcover install and run?

Do I need admin rights to install and configure the licenese?

I use Hudson for my build server so I need to deploy dotcover on the fly (some sort of xcopy), get it configured and executing on machines that sit on a farm.

Is this possible?

Eladio
Hadi Hariri says:
June 3, 2011
@Eladio,

TeamCity includes dotCover already. If you’re using something different (Hudson) in your case, unfortunately right now the only way to do it is installing it which requires Visual Studio. We’re working on removing this requirement.
Ash says:
July 29, 2011
How do I run tests with coverage from the command line for a silverlight app?
Gerhard says:
February 17, 2012
Is it still a requirement to install dotCover on our BuildMachine ? This would be really show stopper … We are also using Hudson.

Thx for a short answer.
Gerhard says:
February 17, 2012
Sorry. I wanted to say – to install VS2010 on our build machine…
Hadi Hariri says:
February 17, 2012
You need to install @dotCover on the build machine to run it still, yes.
DotCoverEvaluator says:
March 29, 2012
Hi,
I am trying to set up DotCover to run it from outside VS.
I am using nunit as unit test framework.

I followed the steps explained in this page, but I get an error message (There is no Application Shell running.) when I run dotCover analyse coverage.xml

———-
$ dotCover analyse coverage.xml
JetBrains dotCover Console Runner v1.2.384.4. Copyright (c) 2009-2012 JetBrains s.r.o. All rights reserved.

Unhandled exception: There is no Application Shell running.
———–
What have I missed?
I have a trailed version of dotCover now.

Best regards!
DotCoverEvaluator says:
March 29, 2012
Hi,
I managed to run it now, but have another problem
But still some error, I get the following message

.
.
Analysed application exited with code ‘-1′

and when I look at the generated html the coverage is 0% while if I run it from VS I have 5%.
What does code ‘-1′ means ?

Thanks!
lukas says:
November 6, 2012
Hey,

is it possible to have a full file path in the xml report? Ex. File foo/boo/moo.cs covered in 60%, so not the class and namespace but a file path ?
srikara says:
November 26, 2012
Is it still a requirement with dotCover 2 to install Visual Studio in the server to get code coverage through console? Or is there a way I can get code coverage with out installing Visual Studio?
Subith says:
January 9, 2013
Hi,

I have installed dotcover 2.2 in VS2012. Ad created a coverage.xml as follows

C:Program Files (x86)JetBrainsdotCoverv2.2BindotCover.exe
E:SVNReposcscr-toolstrunksrcvstudioBOMSystemTestBOM-TestbinDebugBOM_Test.dll
E:SVNReposcscr-toolstrunksrcvstudioBOMSystemTestBOM-TestbinDebug
out.xml

I ran the following commandline
“C:Program Files (x86)JetBrainsdotCoverv2.2BindotCover.exe” analyse E:SVNReposcscr-toolstrunksrcvstudioBOMcoverage.xml

And i am getting thee following error

Unknown command: E:SVNReposcscr-toolstrunksrcvstudioBOMSystemTestBOM-TestbinDebugBOM_Test.dll
Type ‘dotCover help’ for usage.

BOM_Test.dll is my unitest library . Can anyone please hep me to fix this issue?

Thanks,
Subitha
Gopinath says:
February 19, 2013
I have installed DotCovere 1.2 version on my machine and it’s also showing in VS 2010. when i ran the tests, the tests passed successfully but i was not able to find Dot Cover coming into scene anywhere . Do i need to configure it properly in VS 2010 . how can i do that without using Re-sharper .
Gopinath says:
February 19, 2013
How can i generate cover reports in VS 2010 using Dot cover . I have the project and the tests ready .
Lex says:
March 13, 2013
I am evaluating dotcover for use with Windows 8 store Apps solution.
The solution supports a framework, that supports multiple windows 8 store apps.

I attempted a command line run but had no success using vstest.console.exe as the binary.
coverage.xml

vstest.console.exe
C:artifactsUnitTest_BaseCommunication.Tests_25.0.0.0_AnyCPU_TestBaseCommunication.Tests_25.0.0.0_AnyCPU.appx /InIsolation
C:artifactsUnitTest_

Is this possible? Its really unclear to me with current level of documentation.
Prakash Moharer says:
September 9, 2013
Hi Guys, I need a bit of assistance. The issue that I am facing is while running Individual Unit/Integration Test Cases it is passing, but it failed while it is run using dotCover. Please advise
Maarten.Balliauw says:
September 11, 2013
Hi Prakash,

I will ping you via e-mail.

Best regards
Maarten
KZSun says:
October 3, 2013
If for some reason we are using both NUnit Framework and Visual Studio MS Test Framework in the same test project. Then how should I add both Executables?

NUnit Framework Exe

MSTestExe

Thx in advance
nuno says:
November 21, 2013
Is there any way to run the console app without launching the UI? I have dotcover installed on our build machine, and i need a console app that runs silently and deposits the xml output for Jenkins to parse. Initially, i’d like to make the report available to devs from the Jenkins console, but eventually, i’d like to break the build if the coverage dipped below a certain threshhold percentage. Is any of this possible?
Maarten.Balliauw says:
November 21, 2013
Hi Nuno,

You can use the approach outlined in this blog post to run dotCover on the console (it should not show the UI). The tool can output HTML reports or XML reports, depending on hwo you want them to surface in Jenkins. More information can be found in https://www.jetbrains.com/dotcover/webhelp/dotCover__Server_Test_Coverage.html

If you want the build to fail whenever a coverage threshold isn’t met, you will currently have to parse the XML report for that data. Do vote for this feature request http://youtrack.jetbrains.com/issue/DCVR-6223 to make this an easier task.

Feel free to send me an e-mail if you need assistance with this.

Best regards,
Maarten
Patric Pillai says:
February 18, 2014
Hi,
I would like to specify the code coverage parameters for a web application in an XML file to be used with the analyse command in the command console. Could anyone help me on the same.

Thanks

Patric
- Hadi Hariri says:
  February 22, 2014
  Patric, it depends what type of web application. Best option in this case is to contact support@jetbrains.com and they can help you since settings vary.
Vladimir says:
March 18, 2014
What should i do to take coverage of my webapp (IIS)?
Yogesh says:
June 19, 2015
dotCover02_000>dotCover.exe c coverage.xml
JetBrains dotCover Console Runner 3.1.1. Build 20150521.1317
Copyright (c) 2009-2015 JetBrains s.r.o. All rights reserved.
[JetBrains dotCover] Coverage session started [6/19/2015 5:20:06 PM]
Microsoft (R) Test Execution Command Line Tool Version 12.0.21005.1
Copyright (c) Microsoft Corporation. All rights reserved.

Invalid switch “d”.
For switch syntax, type “MSTest /help”
[JetBrains dotCover] Coverage session finished [6/19/2015 5:20:13 PM]
[JetBrains dotCover] Analysed application exited with code ‘1’
[JetBrains dotCover] Coverage results post-processing started [6/19/2015 5:20:13 PM]
[JetBrains dotCover] Coverage results post-processing finished [6/19/2015 5:20:14 PM]

How can i solve it?
Eldad says:
August 28, 2017
Hi,

How can I add multiply arguments from several dll.
Gopi says:
August 30, 2017
We’re running dotcover on Teamcity. While setting coverage filters there’s no way to include or exclude at the namespace level. It either picks an assembly or not. Is there a way to extend this to namespaces?
Alok says:
April 3, 2020
I am getting following error –
[JetBrains dotCover] Analyzed application exited with code ‘2’
I am using nunit-cosole.exe as test runner, and apart from this I am getting many other exit codes like ‘952’, ’45’ etc.
Can you help me with the details of these exit codes?
- Alexander Kurakin says:
  April 3, 2020
  Please file a support request here https://dotnettools-support.jetbrains.com/hc/en-us.

Running Code Coverage from the Console with dotCover

Console Runner

Coverage in one command

Filtering from the Console

Giving it some style

32 Responses to Running Code Coverage from the Console with dotCover

Joe White says:

hhariri says:

Ruslan Isakiev says:

Lokesh says:

Hadi Hariri says:

Eladio says:

Hadi Hariri says:

Ash says:

Gerhard says:

Gerhard says:

Hadi Hariri says:

DotCoverEvaluator says:

DotCoverEvaluator says:

lukas says:

srikara says:

Subith says:

Gopinath says:

Gopinath says:

Lex says:

Prakash Moharer says:

Maarten.Balliauw says:

KZSun says:

nuno says:

Maarten.Balliauw says:

Patric Pillai says:

Hadi Hariri says:

Vladimir says:

Yogesh says:

Eldad says:

Gopi says:

Alok says:

Alexander Kurakin says:

Tags

Latest Tweets

Related posts

Webinar – OOPs, I did it again

Xamarin, the best way to make NFC Apps – Webinar recording