ReSharper, .NET Core and unit testing

Matt Ellis

In this post, we’re going to look at how ReSharper 2016.3 can help you unit test your .NET Core applications and libraries.

Sadly, before we can look at this, we have to deal with the confusing issue of what versions of .NET Core are supported.

TL;DR: ReSharper 2016.3 works with .NET Core projects in Visual Studio 2015 and Visual Studio 2017 “RC”, but unit testing only works in project.json based projects. The IDE interface to dotnet test was rewritten for the .csproj based projects in Visual Studio 2017 “RC” and only just documented. More details at the end of this post.

Unit testing in .NET Core

Right, let’s take a look at unit testing a .NET Core app or library.

The first thing we need to do is get our project set up for unit testing. This involves two things – not only do we need to add a reference to a test framework, but we have to specify the test runner. In this example, we’ll use xUnit.net, but ReSharper also supports NUnit and MSTest. Your project.json should look something like this:

{
  "version": "1.0.0-*",
  "dependencies": {
    "System.Runtime.Serialization.Primitives": "4.3.0",
    "xunit": "2.2.0-*",
    "dotnet-test-xunit": "2.2.0-*"
  },
  "testRunner": "xunit",
  "frameworks": {
    "netcoreapp1.1": {
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.1.0"
        }
      }
    }
  }
}

Note that we have two xUnit dependencies – one for the actual framework, xunit, and another for the test runner itself, dotnet-test-xunit. These dependencies are enough to get our code compiling – the [Fact] attribute will be defined – but no tests will be discovered or run, either by ReSharper or dotnet test.

We must also set the testRunner property to xunit. Once we’ve done that, dotnet test now knows how to talk to the test runner, and ReSharper will be able to discover tests.

Test discovery

The way ReSharper discovers tests mirrors how it works with .NET Framework projects – tests are discovered by analysing a source file and adding markers to the editor gutter and actions to the Alt+Enter menu.

For traditional .NET Framework projects, when a project is compiled, the just-built assembly is further analysed and used as the “source of truth” for the list of tests that will be run – this makes sure that ReSharper is running the tests that are actually available in the assembly.

In .NET Core projects, ReSharper calls dotnet test to perform the same discovery. Any additional tests that are discovered here that weren’t found in the source are added to the Unit Test Explorer and any Unit Test Sessions.

Running and debugging tests

Tests can be run just as you’d expect from ReSharper – with keyboard shortcuts, toolbar buttons in the Unit Test Explorer, and Alt+Enter on the test class and method itself.

To actually run the tests, ReSharper again invokes dotnet test, but in a “design time” mode that allows IDEs to communicate and control the process via a JSON protocol. This means that ReSharper is running your tests in exactly the same way that dotnet test does, but with the benefits of ReSharper’s own user interface and reporting.

All of the usual ReSharper functionality is available here – reporting progress, showing output, filtering to failed tests, exceptions with clickable links to take you directly to classes and methods, different grouping options and so on.

A quick note on test output – if you’re using xUnit, there is a bug in the current implementation of the runner (2.2.0-preview2-*) that means that test output is only sent to ReSharper if the test fails. This is fixed in preview3. (Also remember that Console.WriteLine doesn’t capture output in xUnit, due to parallel execution of tests. See the docs for more info.)

ReSharper also supports debugging .NET Core tests, just like traditional .NET Framework projects – just set the breakpoint and select “Debug test”.

Code coverage and profiling

If you have ReSharper Ultimate installed, you’ll also have dotCover and dotTrace for code coverage and performance profiling, respectively. Both tools integrate nicely with ReSharper’s test runner, adding new options to “Cover test” and “Profile test”. And of course, these work with .NET Core projects, too.

There are a couple of rough edges, however, as we continue to build support for .NET Core.

As we noted when we announced dotTrace support for CoreCLR back in 2016.3 EAP6, .NET Core projects only work in Sampling mode. Unfortunately, dotTrace will prompt you to choose between all profiling modes. If you select anything other than Sampling, the test run fails, and it looks like the test run has failed. Make sure you use Sampling!

Also, dotTrace doesn’t yet support the new “portable” style .pdb debug information files. While profiling will work correctly, selecting a method in the results will be unable to open the associated source file. Setting the debugType property in project.json to full, or leaving it unspecified will work as expected.

Code coverage also has an issue with portable .pdb files – if specified, you might see an error “Coverage analysis failed” and not get any results. If this is the case, please make sure that the “Include not loaded solution assemblies into coverage results” is unchecked in ReSharper → Options → dotCover → General, or use the “full” debug type.

We’re aiming to fix these issues as part of ReSharper 2017.1.

Multiple target frameworks

One of the distinguishing features of .NET Core projects is the ability to target multiple frameworks with a single project file, so the same project file and source code can be used to target the CoreCLR (maybe cross platform) or .NET Framework. Doing this is as simple as adding a couple of extra lines to the frameworks property of project.json:

{
  "version": "1.0.0-*",
  "dependencies": {
    "System.Runtime.Serialization.Primitives": "4.3.0",
    "xunit": "2.2.0-*",
    "dotnet-test-xunit": "2.2.0-*"
  },
  "testRunner": "xunit",
  "frameworks": {
    "netcoreapp1.1": {
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.1.0"
        }
      }
    },
    "net461": {
    }
  }
}

From the command line, dotnet test will now compile and test the project twice, once for CoreCLR, and once for the .NET Framework. ReSharper fully supports multiple target frameworks, too.

Given a project with multiple target frameworks, ReSharper will maintain a separate tree of test classes and methods for each target framework, rather than a single tree of tests. This is for a couple of reasons.

Firstly, it means it’s possible to run tests for all frameworks at once, and show different results for a given test in different frameworks – your test might pass under CoreCLR but fail on the .NET Framework!

Secondly, it makes it a lot easier to run a specific test for a specific target framework, straight from the Unit Test Explorer or Unit Test Sessions windows.

And finally, thanks to #defines, there might actually be a different set of tests available to each framework!

So how does ReSharper work with multiple frameworks? What happens when I do Alt+Enter → Run on a test?

Visual Studio includes a drop down navigation bar at the top of each file, which contains classes and methods. If a project has multiple target frameworks, it also includes the current target framework. Changing this drop down changes the context for the file.

ReSharper already understands this context, and uses it to resolve code with the appropriate dependencies, and to grey out code that is excluded due to #defines.

When using Alt+Enter, the normal run and debug methods will run the selected test class or method for the currently selected target framework. ReSharper also adds a submenu to these items to allow to run “in all target frameworks”, which does what you’d expect.

And of course, this also works with coverage and profiling too. Profiling will simply open multiple results windows, with the results of each framework separately.

Coverage also captures the results from all target frameworks. The way it displays those results depends on the state of the “Match coverage results with current project structure” toggle button in the results toolbar. If enabled, the coverage results are merged to a single view of the project structure. Disabling this button splits the coverage results into separate tree views one for each target framework.

Versioning

OK, some more information on versioning. As everyone is aware, the .NET Core runtime is stable, but the SDK and tooling is still very much in development. ReSharper 2016.3 supports .NET Core 1.0 and 1.1 projects in both Visual Studio 2015 and Visual Studio 2017 “RC”. Navigation, editing, refactoring, etc., all work as expected, for both project.json based projects as well as the new, work in progress, .csproj based projects.

However, ReSharper doesn’t support unit testing in .csproj based projects, only project.json projects. For those playing along with SDKs at home, ReSharper supports unit testing for the 1.0.0-preview2-* and 1.0.0-preview2-1-* SDKs. The current 1.0.0-preview4-* SDKs get standard ReSharper functionality, but not unit testing.

This is because the IDE interface to the dotnet test command was rewritten for Visual Studio 2017 “RC” and has only recently been documented and open sourced as the “VS Test Platform”. This meant we couldn’t provide support in ReSharper 2016.3.

This support will be added as the ReSharper 2017.1 EAP progresses, but we have concerns about the JSON protocol for this interface. For example, it is very verbose, which could lead to performance issues for larger projects. Also, because it was designed for Visual Studio, it doesn’t quite meet ReSharper’s requirements (for example, no message when a test is started, only when it completes, and not enough information to accurately map a test back to its class and method).

The biggest issue we face at the moment is that there is no way for ReSharper to launch the test executable itself. This means that we cannot offer code coverage or performance profiling for newer versions of the .NET Core SDK.

Now that the protocol is open source, we hope to be able to work with Microsoft to address some of these issues (and we have a number of issues open), but given how close the launch date is for Visual Studio 2017, we do not realistically expect these changes to be made in the VS2017 timeframe.

While we’re talking about versioning, Microsoft has no plans to back port the .csproj based SDK tooling to Visual Studio 2015. Full support will only be available on VS2017. This leaves ReSharper in an interesting position. ReSharper currently supports Visual Studio 2010, 2012, 2013, 2015 and 2017. If we support VS2015, what do we do about the .NET Core SDK tooling that is an evolutionary dead end? We currently plan to continue providing our current level of support for project.json based projects in forthcoming ReSharper releases. However, we will not be adding new functionality for Visual Studio 2015, only for the official tooling in Visual Studio 2017.

Summary

ReSharper 2016.3 offers great support for unit testing your project.json based .NET Core projects, with support for test discovery, running and debugging, coverage and profiling as well as targeting multiple frameworks. We’re looking forward to offering the same support for Visual Studio 2017 in ReSharper 2017.1.

Subscribe to a monthly digest curated from the .NET Tools blog:

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!