Performance profiling in Continuous Integration: an updated dotTrace plugin to TeamCity

Alexey Totin

A while ago, we introduced the dotTrace plugin to TeamCity. It helps you prevent application performance regression by profiling integration tests on a TeamCity server.

Now we’re releasing the next version of this plugin that brings some significant improvements. Before we move on to the details, let’s recall how the plugin is used for tracking performance in CI builds:

1. Write an integration test that runs some performance-critical functionality in your application.
2. In the plugin, set a performance threshold for this test, which is the execution time value in ms or value taken from previous successful builds. Now, if the test exceeds its threshold, the plugin will fail the build and save the collected performance snapshot to build artifacts.
3. Open the snapshot in dotTrace and find the exact cause of the performance issue.

This simplified workflow description is missing one obvious yet very important step: plugin configuration. The initial plugin required a number of preparatory steps on the build agent side: installing the console unit test runner and creating a rather complex XML configuration file (containing various profiling options, like profiling target, type, and so on). The new release addresses this issue: you no longer need to install the runner or create the configuration file. The trick is that the plugin is now made a part of unit testing build steps. Thus, after installing the plugin, all these build steps (Visual Studio Tests, NUnit, MSpec, and others) get the additional option Run build step under dotTrace profiler.

Moving profiling under unit testing build steps is also beneficial in one more way. From now on, when the plugin is enabled, you can not only check performance of tests but also track test results as usual (it’s still a “normal” unit testing build step).

Below, we’ll provide the detailed plugin usage example that you can use as a step-by-step guidance for the new dotTrace Plugin for TeamCity.

1. Install the plugin and dotTrace Console Profiler

IMPORTANT! If you use the previous plugin version, uninstall it before proceeding to this step.

1. On your TeamCity server, download and copy dotTrace.zip to the plugins directory located in your TeamCity data directory. Use guest / guest credentials for download.
2. Restart the TeamCity Server service.
3. As the dotTrace console profiling tool is required for the plugin, download dotTrace Command Line Tools and unzip to any directory on a TeamCity build agent.

2. Write an integration test

Suppose we have an application with a killer feature called Foo. For example, we have a Foo class with the Foo() method which, in turn, uses a Goo class. It creates a Goo instance and runs its Goo() method, which also runs some methods. We assume the execution time of the Foo feature is crucial for user experience. So we add a performance NUnit* test that runs the feature:

* The plugin supports all TeamCity .NET unit test runners, but we will use NUnit in our example.

namespace IntegrationTests
{
   [TestFixture]
   public class PerformanceTests
   {
      [Test]
      public void TestFooFeature()
      {
         Foo foo = new Foo();
         foo.Foo();
      }
   }
}

3. Create a build configuration

The next step is to create a build configuration dedicated to performance testing in TeamCity.

IMPORTANT! To help ensure consistency of profiling results, you should assign the build configuration that uses the plugin to a specific build agent (a hardware agent is strongly recommended). For further instructions please refer to TeamCity documentation.

Suppose we already have a project called My Project. Let’s add to that project a configuration consisting of two steps: building the application and running performance tests. To save time, let’s proceed right to configuring the second build step.

1. In build configuration settings, go to Build Steps and click Add build step.
   
2. In Runner type, select NUnit. (If you use a different testing framework, you would select the appropriate runner type, e.g. MSTest or Visual Studio Tests.)
3. Set options for NUnit. The main one here is Run tests from — the relative path to the DLL with tests.
4. Select Run build step under dotTrace profiler.
5. Set the following plugin options and then click Save:
   • Path to dotTrace ConsoleProfiler.exe: the path to the directory that you have unzipped dotTrace Command Line Tools into. In our example, it’s C:\Console Profiler
   • Measure type: the type of profiling you want to use. We strongly recommend that you use only the Sampling type as it gives the most realistic time values. Note that the plugin does not support Timeline profiling.
   • Performance snapshot artifacts path: set the path relative to the artifacts folder for storing the collected performance snapshot. Keep in mind that, depending on application complexity, the snapshot may take up hundreds of MB of disk space. We recommend updating your artifact cleanup policy so that after some time TeamCity would delete the snapshot folder from artifacts.
   • Threshold values : specify the list of methods whose performance you want to check. The pattern is Namespace.Class.Method TotalTime OwnTime, where
     • TotalTime is the method’s execution time, including its own time and the time of the method’s call subtree, in milliseconds;
     • OwnTime is the method’s own execution time, in milliseconds.

   (Setting a value to zero will make the plugin ignore the threshold.)

   If we want to check the method’s time against the corresponding time in previous successful builds, we have three options: a) we can take values for comparison from the first successful build, b) take them from the last successful build, or c) compare against the average value calculated for all successful builds. If so, instead of the absolute threshold value in milliseconds, we should use one of the following prefixes:
   a) F – take value from the first successful build,
   b) L – take value from the last successful build, or
   c) A – take average value based on all prior successful builds
   Then, set the tolerance to indicate by how much the time value may be exceeded, as a percentage.

In our example, we want to track the total execution time of the Foo feature, so we add a threshold for the total time of the TestFooFeature() test. F15 means that the value for comparison (with up to 15% tolerance) must be taken from the first successful build. E.g., if during the first successful build dotTrace measures 1000 ms total time for the method, the method’s threshold for all following builds will equal 1150 ms (1000 + 1000*0.15).

In our example, we also want to check the total time of the Goo() method as it strongly impacts the performance of the Foo feature. Checking the method’s execution time during the build simplifies our job: in case of performance issues with Foo, we will know right away if Goo() is the one to blame. As a threshold for the method, we also use the value from the first successful build, plus 15% tolerance.

4. Set a failure condition

If we want the build step to fail when a test exceeds its performance threshold, we should set a failure condition.

1. In Build Configuration Settings, go to Failure Conditions.
2. In Fail build if, select an error message is logged by build runner.

5. Run the build

Now it’s time to run the build! As we decided to use values from the first successful build for the thresholds, the first build won’t fail in any case — it is used only to set the baselines for the TestFooFeature() and Goo() methods. The time values from this build will become the “golden standard” for all subsequent builds.

1. Run the build.
   As TeamCity had no data on the methods’ execution times before the first build, the build passes successfully (the expected values equal 0).
2. Suppose now that someone has edited the Goo() method and made it slower. We’ll emulate this using Thread.Sleep(200) and re-run the build. Now, the test still passes, but the build fails due to an error:
   
   If we now click on the build, we’ll see the following build results:
   
   As we instructed the plugin to save a performance snapshot, it has stored the archive with the snapshot in the artifacts Snapshots folder:
   
   Note that the snapshot consists of a number of files. Now, we can analyze the snapshot and find the exact cause of the performance flaw:
   
3. As all values are reported as TeamCity statistic values, so you can build trend graphs for the measured time values if you like. To do this, open the Parameters tab of any build and switch to Reported statistic values:
   Click View trend for a particular parameter to see its diagram of changes:
   

Summary

As this post has illustrated, the updated dotTrace plugin for TeamCity no longer requires complex configuration, which makes it easier to use. To check it out on your own, please download the plugin (remember to use guest / guest credentials to download). As always, we’d be happy to hear any feedback from you. Feel free to ask any questions in the comments to this post.

Profile with pleasure!

Subscribe to a monthly digest curated from the .NET Tools blog:

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!