Coverage with TeamCity and dotCover with MSTest, NUnit or MSpec

As some of you know, we recently shipped TeamCity 6 which includes, out of the box, a bundled version of dotCover. What this means is that you can now get free coverage for your code easily, and of course even if you’re using the Professional version of TeamCity.  The setup is quite easy if you are using MSTest and NUnit. For MSpec, you need to take a few additional steps.

Using MSTest / NUnit Runners

Normally build files consist of a series of tasks that involve compilation and running of tests. With TeamCity, you can separate some of these steps out into individual TeamCity Build steps, which is what we will be doing in this case (everything that we see here applies to both MSTest and NUnit).

Here is our build.xml (MSBuild) file:

image

As we can see, other than compiling a solution, which in this case consists of the actual application and the test assemblies, not much else going on.

[Note: this could have been done using the SLN as the Runner Type under TeamCity since this example build script does not do much else. In real scenarios however, build scripts do more than just call a solution (in fact normally you’d call projects not solutions)].

In TeamCity, we create a new build project and setup the VCS root. We then add a new Build Step which calls this MSBuild file:

image

Notice that all we are doing here is calling our MSBuild script. No coverage settings yet.

Next thing is to add an additional build step in TeamCity. This time, we are going to call MSTest as opposed to MSBuild:

SNAGHTML1e386e79

SNAGHTML1e3d642d

(the sections cut out are blank).

In the .NET Coverage tool section we select JetBrains dotCover and then  add the assemblies we want coverage for (just the name of the assembly) prefixing them with +: and filtering out those we do not want coverage for with –:.

That’s all there is to it. Once we run the Build, we should now see a new tab with Coverage Reports as well as a new Artifact which contains the Coverage files zipped up.

image

The Code Coverage tab goes into more detail:

image

We can even drill down into individual classes and examine the code coverage:

image

If we are using NUnit instead of MSTest, the only difference is there test runner we select when adding a new Build Step in TeamCity. Instead of MSTest we choose NUnit along with the version:

image

What about MSpec or my Specific Test Runner?

If we are using MSpec or a different test runner that is not supported directly by TeamCity, we can still get coverage reports; we just need to do a little bit of additional configuration. TeamCity has an API which allows us to send it messages when we want to interact with it (this is actually quite a powerful feature but out of scope for this post so please leave a comment if you’d like me to cover it in more detail). We can leverage this API to tell it when to start coverage and where to get the results from.

Here is the build script for MSpec:

image

We have created two targets. The second one (TeamCity) is the one we are interested in. This does a couple of things:

1. The first <Exec> runs all MSpec tests so that we can see the test results inside TeamCity. This is not strictly necessary for Code Coverage but usually build processes do display these results. The –teamcity option we are passing in to MSpec is for it to generate the system messages that are then fed to TeamCity (see point 3)

2. The second <Exec> is the one that runs dotCover. This uses a configuration file called dotCover.xml which we will examine further down. We pass in the c (or coverage) option when calling dotCover.

3. This is a message we send to TeamCity to tell it that we have run coverage. We indicate the tool we are using (dotcover in this case) and where the results are located. TeamCity uses this information to then display the results in the UI. This is one of the API messages mentioned earlier.

Finally we need to define the dotcover.xml file with out configuration for running dotCover (for detailed information on creating dotCover configuration files, see here and here):

image

In terms of TeamCity, we then just define our build step that calls out to the build script:

image

Notice how we do not specify .NET Coverage options explicitly. And if all goes well, we can see the coverage output just as before:

image

Summary

We can see that running code coverage is now pretty straightforward when using MSTest, NUnit or even a custom test runner. Most of what we have covered for MSpec will work with pretty much any test runner in terms of coverage (feel free to vote here for MSpec support).

With any build process, there are numerous ways of doing the same thing. I’m going to show you one of them. Based on your setup and needs you might want to do things differently. Fortunately TeamCity is flexible enough to allow for many scenarios.

One thing to be aware of is that dotCover creates some temporary files for the XmlSerailizer in the Temp profile folder. This normally is not a problem unless the folder does not exist. If you are running TeamCity under the SYSTEM account, make sure that the folder C:Windowssystem32configsystemprofileAppDataLocalTemp exists. This will probably change in future versions so to avoid any possible issues.

Enjoy!

This entry was posted in News and Events and tagged , , , . Bookmark the permalink.

23 Responses to Coverage with TeamCity and dotCover with MSTest, NUnit or MSpec

  1. Jeremy says:

    Hi Hadi,

    I’ve configured my TeamCity Pro installation as above and although the coverage summary does show up on the overview page for each build no Code Coverage tab exists.

    Is there something specific I need to do to create this tab?

    Thanks!
    Jeremy

  2. Thomas Ardal says:

    Thank you very much for adding the possibility to measure coverage, running tests through msbuild as well (the alternative runners section). Personally I prefer running the tests through msbuild, rather than configuring this as a test task in TeamCity. This way I’m able to recreate the complete build script locally. This is possible with the example above.

  3. Hadi Hariri says:

    @Jeremy,

    Which configuration are you following? What’s your unit test runner? Can you send me an output of the build log (All Messages)? hadi at jetbrains dot com

    Thanks.

  4. I’m using the NUnit runner, and the tests run OK but on the code coverage tab I only get:

    This is an autogenerated index file (there was no index.html found in the generated report).

  5. Hadi Hariri says:

    @Simon,

    Can you look at the Build Log, All Messages? Can you send it to me? (hadi at jetbrains dot com).

    Thanks.

  6. Francesco says:

    Is there a way to use DotConver TeamCity integration with multiple test runners at once in the same project (e.g. MSpec and NUnit)?
    I could create two different projects, but then my coverage results won’t be realistic because they are not merged.

  7. Hadi Hariri says:

    @Franceso,

    TeamCity will merge the reports using dotCover’s merge functionality.

  8. Damian says:

    In your dotcover.xml mspec example the paths for and are absolute. Is that deliberate? Are relative paths supported?

  9. Brett says:

    Did you have to do any post installation to get MSTest 10 to work? I just installed TeamCity 6 and when I add a build step for MSTest with the default MSTest 10 directory specified as you have above I get an agent error stating that MSTest does not exist.

  10. Hadi Hariri says:

    @Brett,

    No, didn’t have to do anything else. Does it work if you add MSTest to the path on the build agent?

  11. Fredrik says:

    Question – can we run coverage for both Release / Debug builds on all platforms (MSTests famously don’t do so well on x64)

  12. Hadi Hariri says:

    @Frederik,

    You should be able to .

  13. Tom Cabanski says:

    Tried to get this working with MSpec and it seems as if the article glosses over the detail of how to get dotCover to actually work. There are two issues I had to take care of before it worked:

    1) dotcover.exe does not end up in the path of the agent automatically and there does not seem to be a TeamCity variable that contains the full path.

    Therefore, I used the following as in the definition of my dotCover property:

    $(agent_home_dir)pluginsdotCoverbindotCover.exe

    2) dotCover.xml needs to have absolute paths. This was mentioned in an earlier comment and I guess should be fixed soon. Until then, MSBuild community tasks let me do this (I have a file called dotCoverTemplate.xml containing the configuration template):

    Note that this all assumes you want to put the dotcover.xml in the root of your project folder as I did.

  14. Hadi Hariri says:

    @Tom,

    Sorry, I should have pointed that out as you said. I had dotCover in the path.

    Regarding the relative paths, also. It was an issue in dotCover 1.0 and is now resolved for 1.1 (in beta), and TeamCity 6.5 (in EAP) ships with 1.1, so that should be solved. In fact 6.5 ships with MSpec support so most of this is now redundant.

    My apologies though on not making those points clear.

  15. DavidS says:

    Hi,

    I believe that the stable version of 6.5 is planned to be released at the end of this month beginning of next month. Is that still the case? Also, how easy will it be to update from 6.5EAP to the stable version?

    TIA,

    David

  16. Hadi Hariri says:

    @DavidS,

    It should be a smooth upgrade.

  17. Dina says:

    Hi,

    I have many build steps in my project, each running unit tests of a different DLL. I have configured DotCover for all of them. But it looks like the results displayed are only for the last build step. When I removed DotCover from the last step and ran my project again, I saw the coverage result for the one before last build step.

    How can the results be merged?

    Thanks,
    Dina

  18. Maarten.Balliauw says:

    @Dina,

    TeamCity will merge coverage results from all build steps in this case. The problem here is probably with the configuration itself. Currently dotCover doesn’t support different filters during snapshots merge
    (we expect this to be fixed in 2.5). As a workaround, you can use less strict filters in every build step.

    Another option is to revert to using just one build step which runs unit tests for specified assemblies. This can be done using wildcards or by adding multiple entries to the include or exclude assemblies list in the build step. For example, to include all assemblies in the “sources” folder which end in .Tests.dll and are in the “bin” folder, you can specify “Sources**bin***.Tests.dll”. Excluding can be done as well using the same approach.

    For coverage, you can also specify which assemblies to include or exclude from coverage. For example if you want to run coverage on all Foo.Bar.* assemblies but not on Foo.Bar.*.Tests, specify the following two assembly filters for coverage:
    +:Foo.Bar.*
    -:Foo.Bar.*.Tests

  19. Dina says:

    @Maarten.Balliauw,

    Thanks! I changed the filters and got all the coverage results.

    I have another question: can I run coverage on a regular console application. I have unit tests that run as a simple console application (actually as nunit console-runner).
    I configured them to run as a MSBuild step and checked the dotCover option. The tests ran successfully but I didn’t get any coverage results.

    Thanks again,
    Dina

  20. Maarten.Balliauw says:

    @Dina,

    If you configure coverage via TeamCity UI, there currently is no way around using the TeamCity test runner (see http://confluence.jetbrains.net/display/TCD8/Configuring+.NET+Code+Coverage#Configuring.NETCodeCoverage-Configuring.NETCodeCoverage-coverageSupportNAntMSBuild). You can also start code coverage manually and report coverage back to TeamCity via service messages (more on http://confluence.jetbrains.net/display/TCD7/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-Reporting.NETCodeCoverageResults)

  21. Caio Proiete says:

    Is there any documentation on how to have DotCover + TypeMock + NUnit working properly on TeamCity?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">