Test .NET Core projects with TeamCity

Greetings, everyone!

As many of you already know, TeamCity has a plugin to build .Net Core projects, which is basically a wrapper for the dotnet command. But if you try running dotnet test, you won’t see test results in TeamCity at the moment. As the test time can be significant, it is preferable to have fast feedback, so in this post we’ll explain how to integrate the dotnet test command with TeamCity and configure on-the-fly test reporting.

In short, you’ll need to perform the following:

  1. To run tests locally using the dotnet test command,  add references to the following packages to your test project: to Microsoft.NET.Test.Sdk, to the test framework, and the test adapter.
  2. To configure on-the-fly reporting, add a package reference to the TeamCity VSTest Adapter.
  3. To configure a build in TeamCity, use the TeamCity plugin for .NET Core projects or some other build runner.

Let’s go over these points in detail.

Run tests locally using dotnet test command

The approach suggested by Microsoft works fine for any target framework as well as for multiple frameworks at the same time, provided the test engine has a test adapter, e.g. MS tests, xunit tests, or some other test engine, for example.

That means that a test project needs to have at least three additional package references:

  • Microsoft.NET.Test.Sdk, which contains the MSBuild targets and properties for test projects and marks a project as a test project.
  • The selected Test Framework, e.g.  MSTest, or XUnit, or other
  • Test Adapter to discover and execute test framework based tests, e.g. the MSTest adapter, or the XUnit adapter, or other.

Configure on-the-fly reporting in TeamCity

Ideally, you should not need to do any additional preparation steps if you choose to run dotnet tests on TeamCity: TeamCity should pick up these tests automatically. For now this is not the case, so here is how you still can achieve this goal: use the TeamCity.VSTest.TestAdapter NuGet package containing a special logger which integrates with the test platform and sends service messages to a TeamCity server.

To turn on the integration, you’ll have to add a package reference to TeamCity VSTest Adapter. Note that this package does not impact the tests run locally, but only those run under TeamCity.

Here is an example of a project:

Visual Studio NuGet Dependencies

The project file for an MS test project can look as follows:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>    
    <TargetFrameworks>net45;netcoreapp1.0</TargetFrameworks>    
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="1.1.14" />
    <PackageReference Include="MSTest.TestFramework" Version="1.1.14" />
    <PackageReference Include="TeamCity.VSTest.TestAdapter" Version="1.0.0" />    
  </ItemGroup>

  <ItemGroup>
    <Service Include="{82a7f48d-3b50-4b1e-b82e-3ada8210c358}" />
  </ItemGroup>

</Project>

Configure build in TeamCity

To configure a build project in TeamCity, we used the TeamCity plugin for .NET Core projects, described here in detail.

You can see an example on the public TeamCity server, where a test project contains one class, UnitTests:

namespace MS.Tests
{
    using System;
    using Microsoft.VisualStudio.TestTools.UnitTesting;

    [TestClass]
    public class UnitTests
    {
        [TestMethod, Ignore]
        public void TestIgnored()
        {
        }

        [TestMethod]
        public void TestPassed()
        {
            Console.WriteLine("some text");
        }
    }
}

Here is an example of the TeamCity tests step:

.

After a build is run, TeamCity shows the test results:

This .Net project has two target platforms, net45 and netcoreapp1.0, that is why each test is run for both framework and you can see several results per test.

The proposed approach does require some extra effort, however, it is quite viable and useful for testing .Net Core projects. Currently we are working on simplifying running tests and configuring on-the-fly test reporting from TeamCity, which would not require modifying your project files, so stay tuned for the latest news.

As usual, feel free to share your feedback.

Happy building and testing!

This entry was posted in Features, How-To's, Uncategorized. Bookmark the permalink.

5 Responses to Test .NET Core projects with TeamCity

  1. johnny says:

    Thanks, very useful.

    Can you apply wildcards to the project path like

    **/*.Tests.csproj

    I want to run all tests in multiple projects that contain the work “tests” in the project file name without having to specify every .csproj

  2. NightOwl888 says:

    I want to know the same thing as johnny. Our solution has ~25 test projects and it would be nice to use a glob pattern to locate them.

    I suppose that adding a PowerShell step before this one and putting the result of that logic into an environment variable is possible, but it would be nice to know, is globbing built in?

  3. NightOwl888 says:

    Problem: The project name field always appends “project.json” to the end of the file, even if you specify it as “.csproj”. Microsoft no longer supports “project.json” – why is this plugin using that format?

    Is there an updated version of this that works with .NET Core 2.0/new .csproj format?

  4. Ashleyc says:

    I have a .NET CLI (dotnet) test Build Step failing and yet subsequent build steps are executed.

    When I manually run the test step from the command line it exits with a code of 1.
    The Failure Conditions for my build configuration have the following selected:
    * the build process exit code is not zero
    * at least one test failed
    The subsequent build steps are setup to Execute:
    * If all previous steps finished successfully

    Is it possible the .net runner has an issue similar to
    https://youtrack.jetbrains.com/issue/TW-17002

    Here is the Build log output
    [13:40:04]Step 4/8: Run Tests (.NET CLI (dotnet)) (2s)
    [13:40:04][Step 4/8] Starting: “C:\Program Files\dotnet\dotnet.exe” test C:\\BuildAgent\\work\\a5daf7c37b533830\\Blah\\Blah.csproj –no-build @C:\\BuildAgent\\temp\\buildTmp\\41ee3ffafd994f5cb7642ee601c41319.rsp
    [13:40:04][Step 4/8] in directory: C:\BuildAgent\work\a5daf7c37b533830
    [13:40:05][Step 4/8]
    [13:40:05][Step 4/8] Build succeeded.
    [13:40:05][Step 4/8] 0 Warning(s)
    [13:40:05][Step 4/8] 0 Error(s)
    [13:40:05][Step 4/8]
    [13:40:05][Step 4/8] Time Elapsed 00:00:00.59
    [13:40:05][Step 4/8]
    [13:40:05][Step 4/8] Test run for C:\BuildAgent\work\a5daf7c37b533830\Blah\bin\Debug\netcoreapp2.0\Blah.dll(.NETCoreApp,Version=v2.0)
    [11:02:25][Step 4/8] Microsoft (R) Test Execution Command Line Tool Version 15.3.0-preview-20170628-02
    [11:02:25][Step 4/8] Copyright (c) Microsoft Corporation. All rights reserved.
    [11:02:25][Step 4/8]
    [11:02:25][Step 4/8] Starting test execution, please wait…
    [11:02:27][Step 4/8] Blah.dll
    [11:02:27][Blah.dll] Blah.UnitTest1.TestMethod1
    [11:02:27][Step 4/8] Failed Blah.UnitTest1.TestMethod2
    [11:02:27][Step 4/8] Error Message:
    [11:02:27][Step 4/8] Assert.AreEqual failed. Expected:. Actual:.
    [11:02:27][Step 4/8] Stack Trace:
    [11:02:27][Step 4/8] at Blah.UnitTest1.TestMethod2() in C:\BuildAgent\work\a5daf7c37b533830\Blah\UnitTest1.cs:line 17
    [11:02:27][Step 4/8]
    [11:02:27][Blah.dll] Blah.UnitTest1.TestMethod2
    [11:02:27][Blah.UnitTest1.TestMethod2] Assert.AreEqual failed. Expected:. Actual:.
    [11:02:27][Blah.UnitTest1.TestMethod2] at Blah.UnitTest1.TestMethod2() in C:\BuildAgent\work\a5daf7c37b533830\Blah\UnitTest1.cs:line 17
    [11:02:27][Step 4/8]
    [11:02:27][Step 4/8] Test Run Failed.
    [11:02:27][Step 4/8] Total tests: 2. Passed: 1. Failed: 1. Skipped: 0.
    [11:02:27][Step 4/8] Test execution time: 1.8562 Seconds
    [11:02:27][Step 4/8]
    [11:02:27][Step 4/8] Build FAILED.
    [11:02:27][Step 4/8] 0 Warning(s)
    [11:02:27][Step 4/8] 0 Error(s)
    [11:02:27][Step 4/8]
    [11:02:27][Step 4/8] Time Elapsed 00:00:04.74
    [11:02:27][Step 4/8]
    [11:02:27][Step 4/8] Process exited with code 1
    [11:02:27]Step 5/8: Publish (.NET CLI (dotnet)) (9s)
    [11:02:27][Step 5/8] MSBuild Response File

Comments are closed.