Shadow-copying in dotCover: if your NUnit tests fail during continuous testing

When dotCover 10 came out, early users of continuous testing faced a problem: tests that used external files could fail in a continuous testing session while being successful in a ‘normal’ unit testing session. The issue was related to how dotCover worked with unit test assemblies.

To run a continuous testing session in the background, dotCover doesn’t use the original unit test assembly: instead, it creates a copy using the shadow-copying mechanism. Along with assemblies and symbol files, dotCover 10 and 10.0.1 also copied the entire content of the directory where the original assembly was located, be it a default location (e.g., …/bin/Release/) or a custom location specified by the option ReSharper | Options | Unit Testing | Run tests from.

If a unit test works with a file (using its relative path) that is located outside this directory, the test will fail as it looks for the file relative to the new temporary location. For example, the following test (used to process a file in some way) fails in a continuous testing session but passes in a ‘normal’ unit testing session.

Failed test in a continuous testing session

Successful test in a normal session

To address the issue, dotCover 10.0.2 introduced a new option, ReSharper | Options… | dotCover | Continuous Testing | Shadow-Copy.

Shadow-copy option

If this option is set to Assemblies and symbol files, dotCover will copy to a temporary location only assemblies and PDB files without touching other data. In this case, the working directory for tests will be the initial location of test assemblies (e.g., …/bin/Release/). Therefore, the sample test from above will pass successfully in both continuous testing sessions and traditional unit testing sessions.

Successful test in a continuous testing session

However, if All files from the tests start folder is selected, dotCover works as before. It copies the entire content of the tests’ start directory. In this case, the path ..\\..\\Test Files\\1.txt will refer to the new temporary location, causing the test to fail.

IMPORTANT! The Shadow-Copy option works as described ONLY with NUnit tests. All other unit testing frameworks use shadow-copying at their discretion depending on plugin implementation. For example, MSTest may decide to disable shadow-copying in a particular case (regardless of the Shadow-Copy option).

To simplify using continuous testing, you may want to change the way you reference files in tests. Here is a short table showing how you can build paths to the tests working directory. Source here refers to the initial directory where the unit test assembly is located; Temp means the temporary directory where the assembly copy is located.

Assemblies and symbol files All files from the tests start folder
Assembly.GetExecutingAssembly().Location Temp Temp
Assembly.GetExecutingAssembly().CodeBase Source Temp
Environment.CurrentDirectory Source Temp
AppDomain.CurrentDomain.BaseDirectory Source Temp
TestContext.CurrentContext.TestDirectory (NUnit 2.6) Source Temp
TestContext.CurrentContext.WorkDirectory (NUnit 2.6) Source Temp
This entry was posted in How-To's and tagged , , , . Bookmark the permalink.

2 Responses to Shadow-copying in dotCover: if your NUnit tests fail during continuous testing

  1. Mike-EEE says:

    This is a very informative and important article! I recently submitted a bug that I was experiencing with CT, and now I am sure it is due to this issue:
    https://youtrack.jetbrains.com/issue/DCVR-7978

    Although, I am using xUnit so I am not sure if that supports shadow-copying. FWIW, I did look into the shadow copying setting in the Tools -> Unit Testing and thought that might be the issue, but obviously that did not address it. I didn’t realize there was a CT setting in dotCover right below that. Thoroughness fail on my part. 😛

    One thing I would note is that it does appear that there is some disconnection between unit testing, CT, and the rest of the ReSharper ecosystem that needs some smoothing out. For instance, I ran into an issue where dotTrace was not able to find pdb files because of shadow copying:
    https://youtrack.jetbrains.com/issue/DTRC-22927

    CT and UT are treated separately, and in my mind (at least) I feel they should be one and the same, with CT as a subset/subfeature of UT. I have a couple issues filed in YouTrack regarding this (in case someone else wants to see and vote for these issues):
    https://youtrack.jetbrains.com/issue/DCVR-7941
    https://youtrack.jetbrains.com/issue/DCVR-7935

    In any case, CT has completely changed my development world. I believe in its vision and power. :) I have been using CT along w/ xUnit and AutoFixture for the past several weeks and it has easily been the best time in my career as far as QA goes. SO MUCH POWER!!! Really a sweet spot in testing, Team ReSharper! Thank you for making this incredible product and feature.

  2. Pingback: Dew Drop – January 4, 2016 (#2160) | Morning Dew

Leave a Reply

Your email address will not be published. Required fields are marked *