Advanced scenarios with dotCover Console Runner

In an earlier post, we saw how to use dotCover in simple scenarios, whereby we have a single unit test project for which we want to obtain coverage reports on. There are times however, when this is not the case; that is, we might have multiple tests projects. It is very common (and recommended practice) to separate Unit test projects from Integration test projects, allowing us to run the former more frequently during development and the latter during an Automated Continuous Integration setup. In these cases, running the dotCover analyse command, as shown in the previous post might not be ideal.

Fine tuning the coverage steps

For the scenarios described, whereas we have multiple test projects, dotCover allows us to fine-tune the coverage process by offering us a series of commands that breakdown the coverage, merging and reporting into individual steps. Suppose we have the following Project Layout

image

The AppCode is our application code that we want to run coverage for, and TestProject1 and TestProject2 are specific the test projects that cover AppCode. These can be Unit Test or Integration Test projects. The process we need to follow is shown in the figure below:

image

We run coverage on each of the tests projects by using the cover (c) command using the following configuration file (corresponds to testProject1.xml. testProject2.xml is similar)

<?xml version=”1.0” encoding=”utf-8“?>
<CoverageParams>
<Executable>
X:LibrariesnUnitnunit-console.exe
</Executable>
<Arguments>TestProject1.dll</Arguments>
<WorkingDir>TestProject1bindebug</WorkingDir>
<Output>output1.xml</Output>
<Filters>
<IncludeFilters>
<FilterEntry>
<ModuleMask>AppCode</ModuleMask>
<ClassMask>*</ClassMask>
<FunctionMask>*</FunctionMask>
</FilterEntry>
</IncludeFilters>
<ExcludeFilters>
</ExcludeFilters>
</Filters>
</CoverageParams>

[Command to execute: dotcover c testProject1.xml]

Executing this command generates a snapshot file with coverage data, the Coverage Results Descriptor (by default, unless specified in the configuration file using the TempDir element, the snapshots are saved in the User temporary folder). These snapshots are not the final XML report that we are after but intermediate information that dotCover uses to generate reports from. There is a correspondence between snapshots and processes, such that each process generates a single snapshot. In our case, since we are calling a unit testing framework to launch a single test project, there will only be one snapshot. However there are cases, for instance when running coverage report on an application, whereby the application can launch another process  itself. In this case, multiple snapshots will be generated.

When we run this command, once for each Project, we end up with two snapshot files. We now need to merge the snapshots in order to generate the XML report. This is accomplished using the merge (m) command

<?xml version=”1.0” encoding=”utf-8“?>
<MergeParams>
<Source>
<string>output1.xml</string>
<string>output2.xml</string>
</Source>
<Output>output_merged.xml</Output>
</MergeParams>

[Command to execute: dotcover m merge.xml]

which takes the output of the two cover commands (output1.xml and output2.xml) and generates a single snapshot file. The only thing remaining is to generate an XML coverage report based off of this data, using the report command

<?xml version=”1.0” encoding=”utf-8“?>
<ReportParams>
<Source>output_merged.xml</Source>
<Output>results_merged.xml</Output>
</ReportParams>

[Command to execute: dotcover r report.xml]

providing us with the XML report containing coverage information (percentage of code covered, statements covered), which like in the previous post we can apply an XSLT to format into HTML or JSON.

Other Coverage Commands

dotCover provides two more commands:

  • list
  • delete

The list command simply obtains a list of all snapshot files given the coverage descriptors, that is, the output of the different cover commands. The configuration file lists these descriptors:

<?xml version=”1.0” encoding=”utf-8“?>
<ListParams>
<Source><!– Required. At least one child element expected. –>
<string><!– Coverage results descriptor 1 file name –></string>
<string><!– Coverage results descriptor 2 file name –></string>
<string><!– Coverage results descriptor N file name –></string>
</Source>
<Output><!– Required. Resulting file name. Stores plain list of all snapshot files. –></Output>
</ListParams>

The delete command deletes the list of given snapshots:

<?xml version=”1.0” encoding=”utf-8“?>
<DeleteParams>
<Source><!– Required. At least one child element expected. –>
<string><!– Coverage results descriptor 1 file name –></string>
<string><!– Coverage results descriptor 2 file name –></string>
<string><!– Coverage results descriptor N file name –></string>
</Source>
</DeleteParams>

Summary

As we can see, dotCover allows us to fine-tune the coverage process by letting us control the different steps. One question that might arise is why couldn’t we have done the same by running the analyse command twice, once for each Test Project. If we were to do that, we would get two independent reports, whereas here the results are all merged into a single report. The analyse command is a great fit for simple scenarios that only have a single project. Once we move onto more complex setups, having a finer control over the coverage process has its benefits.

This entry was posted in dotCover Tips&Tricks, How-To's and tagged , , . Bookmark the permalink.

14 Responses to Advanced scenarios with dotCover Console Runner

  1. JT Keller says:

    I’m having problems doing code coverage in a 2 step process. If I first do a “cover” then a “report” all my code coverage is 0. However, if I just do a “analyse” using the same coverage.xml , but changing to I’m getting valid code coverage. What’s weird too is that when I do the 2 step process the xml report has all the same entries but the code coverage is 0. Have any ideas what I’m doing wrong

  2. Hadi Hariri says:

    @JK

    Are you running report on the same coverage configuration or creating the corresponding report xml file? It’s required for advanced scenarios.

  3. JT Keller says:

    Here is my “Coverage”

    c:Program Files (x86)Microsoft Visual Studio 10.0Common7IDEmstest.exe
    /testmetadata:UnitTest.vsmdi /testlist:TestList0 /testsettings:BuildServer.testsettings /resultsfile:UnitTest.trx
    WhiteboxTestsbinDebug
    CoverageTempUnitTest
    UnitTest.output.xml

    *
    Actuate.Spreadsheet.*
    *

    *Test*
    *
    *

    And here is my “report”

    UnitTest.output.xml
    UnitTest.report.xml

  4. JT Keller says:

    Oops, the bracket characters must have been removed lemme try again…

    Here is my “Coverage”

    &lt?xml version=”1.0″ encoding=”utf-8″?&gt
    &ltCoverageParams&gt
    &ltExecutable&gtc:Program Files (x86)Microsoft Visual Studio 10.0Common7IDEmstest.exe&lt/Executable&gt
    &ltArguments&gt/testmetadata:UnitTest.vsmdi /testlist:TestList0 /testsettings:BuildServer.testsettings /resultsfile:UnitTest.trx&lt/Arguments&gt
    &ltWorkingDir&gtWhiteboxTestsbinDebug&lt/WorkingDir&gt
    &ltTempDir&gtCoverageTempUnitTest&lt/TempDir&gt
    &ltOutput&gtUnitTest.output.xml&lt/Output&gt
    &ltFilters&gt
    &ltIncludeFilters&gt
    &ltFilterEntry&gt
    &ltModuleMask&gt*&lt/ModuleMask&gt
    &ltClassMask&gtActuate.Spreadsheet.*&lt/ClassMask&gt
    &ltMethodMask&gt*&lt/MethodMask&gt
    &lt/FilterEntry&gt
    &lt/IncludeFilters&gt
    &ltExcludeFilters&gt
    &ltFilterEntry&gt
    &ltModuleMask&gt*Test*&lt/ModuleMask&gt
    &ltClassMask&gt*&lt/ClassMask&gt
    &ltMethodMask&gt*&lt/MethodMask&gt
    &lt/FilterEntry&gt
    &lt/ExcludeFilters&gt
    &lt/Filters&gt
    &lt/CoverageParams&gt

    And here is my “report”
    &lt?xml version=”1.0″ encoding=”utf-8″?&gt
    &ltReportParams&gt
    &ltSource&gtUnitTest.output.xml&lt/Source&gt
    &ltOutput&gtUnitTest.report.xml&lt/Output&gt
    &lt/ReportParams&gt

  5. Hadi Hariri says:

    @JT

    And you’re running this once with -c and once with -r? Can you please send me the files so I can look at them better to hadi at jetbrains dot com?

    Thanks.

  6. MV says:

    I got the same problem, and that’s very strange…

    My output result after report :

    <?xml version=”1.0″ encoding=”utf-8″?>
    <Root Name=”Root” CoveredStatements=”0″ TotalStatements=”119″ CoveragePercent=”0″ DotCoverVersion=”1.0″>
    <Assembly Name=”SilverlightTestApplication” CoveredStatements=”0″ TotalStatements=”71″ CoveragePercent=”0″>
    <Namespace Name=”SilverlightTestApplication” CoveredStatements=”0″ TotalStatements=”71″ CoveragePercent=”0″>
    <Type Name=”App” CoveredStatements=”0″ TotalStatements=”38″ CoveragePercent=”0″>
    <Type Name=”<>c__DisplayClass2″ CoveredStatements=”0″ TotalStatements=”2″ CoveragePercent=”0″>
    <Member Name=”.ctor” CoveredStatements=”0″ TotalStatements=”0″ CoveragePercent=”0″ />
    <Member Name=”<Application_UnhandledException>b__0″ CoveredStatements=”0″ TotalStatements=”2″ CoveragePercent=”0″ />
    </Type>
    <Member Name=”.ctor” CoveredStatements=”0″ TotalStatements=”7″ CoveragePercent=”0″ />
    <Member Name=”Application_Exit” CoveredStatements=”0″ TotalStatements=”2″ CoveragePercent=”0″ />
    <Member Name=”Application_Startup” CoveredStatements=”0″ TotalStatements=”3″ CoveragePercent=”0″ />
    <Member Name=”Application_UnhandledException” CoveredStatements=”0″ TotalStatements=”7″ CoveragePercent=”0″ />
    <Member Name=”InitializeComponent” CoveredStatements=”0″ TotalStatements=”7″ CoveragePercent=”0″ />
    <Member Name=”ReportErrorToDOM” CoveredStatements=”0″ TotalStatements=”10″ CoveragePercent=”0″ />
    </Type>
    <Type Name=”Calculator” CoveredStatements=”0″ TotalStatements=”11″ CoveragePercent=”0″>
    <Member Name=”.ctor” CoveredStatements=”0″ TotalStatements=”0″ CoveragePercent=”0″ />
    <Member Name=”Calculate” CoveredStatements=”0″ TotalStatements=”8″ CoveragePercent=”0″ />
    <Member Name=”IsNumeric” CoveredStatements=”0″ TotalStatements=”3″ CoveragePercent=”0″ />
    </Type>
    <Type Name=”MainPage” CoveredStatements=”0″ TotalStatements=”22″ CoveragePercent=”0″>
    <Member Name=”.ctor” CoveredStatements=”0″ TotalStatements=”5″ CoveragePercent=”0″ />
    <Member Name=”ButtonClick” CoveredStatements=”0″ TotalStatements=”3″ CoveragePercent=”0″ />
    <Member Name=”InitializeComponent” CoveredStatements=”0″ TotalStatements=”10″ CoveragePercent=”0″ />
    <Member Name=”SetDefaultValue” CoveredStatements=”0″ TotalStatements=”4″ CoveragePercent=”0″ />
    </Type>
    </Namespace>
    </Assembly>
    <Assembly Name=”SilverlightTestApplication.UnitTests” CoveredStatements=”0″ TotalStatements=”48″ CoveragePercent=”0″>
    <Namespace Name=”SilverlightTestApplication.UnitTests” CoveredStatements=”0″ TotalStatements=”48″ CoveragePercent=”0″>
    <Type Name=”App” CoveredStatements=”0″ TotalStatements=”38″ CoveragePercent=”0″>
    <Type Name=”<>c__DisplayClass2″ CoveredStatements=”0″ TotalStatements=”2″ CoveragePercent=”0″>
    <Member Name=”.ctor” CoveredStatements=”0″ TotalStatements=”0″ CoveragePercent=”0″ />
    <Member Name=”<Application_UnhandledException>b__0″ CoveredStatements=”0″ TotalStatements=”2″ CoveragePercent=”0″ />
    </Type>
    <Member Name=”.ctor” CoveredStatements=”0″ TotalStatements=”7″ CoveragePercent=”0″ />
    <Member Name=”Application_Exit” CoveredStatements=”0″ TotalStatements=”2″ CoveragePercent=”0″ />
    <Member Name=”Application_Startup” CoveredStatements=”0″ TotalStatements=”3″ CoveragePercent=”0″ />
    <Member Name=”Application_UnhandledException” CoveredStatements=”0″ TotalStatements=”7″ CoveragePercent=”0″ />
    <Member Name=”InitializeComponent” CoveredStatements=”0″ TotalStatements=”7″ CoveragePercent=”0″ />
    <Member Name=”ReportErrorToDOM” CoveredStatements=”0″ TotalStatements=”10″ CoveragePercent=”0″ />
    </Type>
    <Type Name=”Tests” CoveredStatements=”0″ TotalStatements=”10″ CoveragePercent=”0″>
    <Member Name=”.ctor” CoveredStatements=”0″ TotalStatements=”0″ CoveragePercent=”0″ />
    <Member Name=”TestCalculator” CoveredStatements=”0″ TotalStatements=”5″ CoveragePercent=”0″ />
    <Member Name=”TestMainWindow” CoveredStatements=”0″ TotalStatements=”5″ CoveragePercent=”0″ />
    </Type>
    </Namespace>
    </Assembly>
    </Root>

    Please, help me !

  7. MV says:

    P.S.: That’s for a Silverlight Project, and I had the same configuration file as JT Killer

  8. Hadi Hariri says:

    @MV

    Can you send me the project files and config to hadi at jetbrains dot com and I’ll take a look? It’s hard to decipher XML on the blog post.

  9. MV says:

    Hi Hadi,

    I had send my project files and config the last week and I’m waiting for your answer.

  10. Hadi Hariri says:

    @MV

    Yes thanks. I’m sorry I’ve not gotten back to you. I’m at TechDays currently and haven’t had a chance to look into it. Will take a look ASAP.

  11. MV says:

    Thank you Hadi, I will wait your feedback. Good TechDays !

  12. Andy says:

    MV. Did you get any feedback on this?

  13. J Dub says:

    I am following the steps in this blog. I have 4 separate test projects. I run “c” on a configuration file for each of the separate projects. This outputs 4 different result files. I then run “m” on a configuration file containing the 4 result files as “Source” strings. The “m” command displays that there are 4 source snapshots. I then run the “r” command on a configuration file that has the merge result file as the Source, and all that I get is a report of the results of 1 of the “cover” runs.

    What am I doing wrong? It seems like maybe the merge command isn’t working for me. Any help would be appreciated.

  14. bsundsbo says:

    Does anyone know if there are any plans to create a tool for dotCover to generate the console configuration file based on your current project? I want to be able to exclude what I see fit in the project from coverage, and then regenerate the project file (includes/excludes).

    That, in my opinion would be quite time efficient.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">