Performance profiling in Continuous Integration: an updated dotTrace plugin to TeamCity

A while ago, we introduced the dotTrace plugin to TeamCity. It helps you prevent application performance regression by profiling integration tests on a TeamCity server.

Now we’re releasing the next version of this plugin that brings some significant improvements. Before we move on to the details, let’s recall how the plugin is used for tracking performance in CI builds:

  1. Write an integration test that runs some performance-critical functionality in your application.
  2. In the plugin, set a performance threshold for this test, which is the execution time value in ms or value taken from previous successful builds. Now, if the test exceeds its threshold, the plugin will fail the build and save the collected performance snapshot to build artifacts.
  3. Open the snapshot in dotTrace and find the exact cause of the performance issue.

This simplified workflow description is missing one obvious yet very important step: plugin configuration. The initial plugin required a number of preparatory steps on the build agent side: installing the console unit test runner and creating a rather complex XML configuration file (containing various profiling options, like profiling target, type, and so on). The new release addresses this issue: you no longer need to install the runner or create the configuration file. The trick is that the plugin is now made a part of unit testing build steps. Thus, after installing the plugin, all these build steps (Visual Studio Tests, NUnit, MSpec, and others) get the additional option Run build step under dotTrace profiler.

Plugin integration

Moving profiling under unit testing build steps is also beneficial in one more way. From now on, when the plugin is enabled, you can not only check performance of tests but also track test results as usual (it’s still a “normal” unit testing build step).

Below, we’ll provide the detailed plugin usage example that you can use as a step-by-step guidance for the new dotTrace Plugin for TeamCity.

1. Install the plugin and dotTrace Console Profiler

IMPORTANT! If you use the previous plugin version, uninstall it before proceeding to this step.

  1. On your TeamCity server, download and copy dotTrace.zip to the plugins directory located in your TeamCity data directory. Use guest / guest credentials for download.
  2. Restart the TeamCity Server service.
  3. As the dotTrace console profiling tool is required for the plugin, download dotTrace Command Line Tools and unzip to any directory on a TeamCity build agent.

2. Write an integration test

Suppose we have an application with a killer feature called Foo. For example, we have a Foo class with the Foo() method which, in turn, uses a Goo class. It creates a Goo instance and runs its Goo() method, which also runs some methods. We assume the execution time of the Foo feature is crucial for user experience. So we add a performance NUnit* test that runs the feature:

* The plugin supports all TeamCity .NET unit test runners, but we will use NUnit in our example.

3. Create a build configuration

The next step is to create a build configuration dedicated to performance testing in TeamCity.

IMPORTANT! To help ensure consistency of profiling results, you should assign the build configuration that uses the plugin to a specific build agent (a hardware agent is strongly recommended). For further instructions please refer to TeamCity documentation.

Suppose we already have a project called My Project. Let’s add to that project a configuration consisting of two steps: building the application and running performance tests. To save time, let’s proceed right to configuring the second build step.

  1. In build configuration settings, go to Build Steps and click Add build step.
    Build configuration
  2. In Runner type, select NUnit. (If you use a different testing framework, you would select the appropriate runner type, e.g. MSTest or Visual Studio Tests.)
  3. Set options for NUnit. The main one here is Run tests from — the relative path to the DLL with tests.
  4. Select Run build step under dotTrace profiler.
  5. Set the following plugin options and then click Save:
    • Path to dotTrace ConsoleProfiler.exe: the path to the directory that you have unzipped dotTrace Command Line Tools into. In our example, it’s C:\Console Profiler
    • Measure type: the type of profiling you want to use. We strongly recommend that you use only the Sampling type as it gives the most realistic time values. Note that the plugin does not support Timeline profiling.
    • Performance snapshot artifacts path: set the path relative to the artifacts folder for storing the collected performance snapshot. Keep in mind that, depending on application complexity, the snapshot may take up hundreds of MB of disk space. We recommend updating your artifact cleanup policy so that after some time TeamCity would delete the snapshot folder from artifacts.
    • Threshold values : specify the list of methods whose performance you want to check. The pattern is Namespace.Class.Method TotalTime OwnTime, where
      • TotalTime is the method’s execution time, including its own time and the time of the method’s call subtree, in milliseconds;
      • OwnTime is the method’s own execution time, in milliseconds.

    (Setting a value to zero will make the plugin ignore the threshold.)

    If we want to check the method’s time against the corresponding time in previous successful builds, we have three options: a) we can take values for comparison from the first successful build, b) take them from the last successful build, or c) compare against the average value calculated for all successful builds. If so, instead of the absolute threshold value in milliseconds, we should use one of the following prefixes:
    a) F – take value from the first successful build,
    b) L – take value from the last successful build, or
    c) A – take average value based on all prior successful builds
    Then, set the tolerance to indicate by how much the time value may be exceeded, as a percentage.

dotTrace build step configuration

In our example, we want to track the total execution time of the Foo feature, so we add a threshold for the total time of the TestFooFeature() test. F15 means that the value for comparison (with up to 15% tolerance) must be taken from the first successful build. E.g., if during the first successful build dotTrace measures 1000 ms total time for the method, the method’s threshold for all following builds will equal 1150 ms (1000 + 1000*0.15).

In our example, we also want to check the total time of the Goo() method as it strongly impacts the performance of the Foo feature. Checking the method’s execution time during the build simplifies our job: in case of performance issues with Foo, we will know right away if Goo() is the one to blame. As a threshold for the method, we also use the value from the first successful build, plus 15% tolerance.

4. Set a failure condition

If we want the build step to fail when a test exceeds its performance threshold, we should set a failure condition.

  1. In Build Configuration Settings, go to Failure Conditions.
  2. In Fail build if, select an error message is logged by build runner.

Build step failure condition

5. Run the build

Now it’s time to run the build! As we decided to use values from the first successful build for the thresholds, the first build won’t fail in any case — it is used only to set the baselines for the TestFooFeature() and Goo() methods. The time values from this build will become the “golden standard” for all subsequent builds.

  1. Run the build.
    Build step first runAs TeamCity had no data on the methods’ execution times before the first build, the build passes successfully (the expected values equal 0).
  2. Suppose now that someone has edited the Goo() method and made it slower. We’ll emulate this using Thread.Sleep(200) and re-run the build. Now, the test still passes, but the build fails due to an error:
    Build step second run
    If we now click on the build, we’ll see the following build results:
    Methods that exceeded performance thresholds
    As we instructed the plugin to save a performance snapshot, it has stored the archive with the snapshot in the artifacts Snapshots folder:
    Performance snapshot in build artifacts
    Note that the snapshot consists of a number of files. Now, we can analyze the snapshot and find the exact cause of the performance flaw:
    Snapshot opened in dotTrace
  3. As all values are reported as TeamCity statistic values, so you can build trend graphs for the measured time values if you like. To do this, open the Parameters tab of any build and switch to Reported statistic values:
    Reported statistic valuesClick View trend for a particular parameter to see its diagram of changes:
    Statistic value trend

Summary

As this post has illustrated, the updated dotTrace plugin for TeamCity no longer requires complex configuration, which makes it easier to use. To check it out on your own, please download the plugin (remember to use guest / guest credentials to download). As always, we’d be happy to hear any feedback from you. Feel free to ask any questions in the comments to this post.

Profile with pleasure!

This entry was posted in How-To's and tagged , , , . Bookmark the permalink.

13 Responses to Performance profiling in Continuous Integration: an updated dotTrace plugin to TeamCity

  1. Pingback: Dew Drop – February 17, 2016 (#2190) | Morning Dew

  2. Pingback: 19-02-2016 - Freaking friday link pack! - Magnus Udbjørg

  3. Sviataslau Seviaryn says:

    Hi!
    Is there any way to set a ‘global’ threshold for all test methods instead of specifying separate values for every test?

  4. Hariharan S says:

    That’s the great way to integrate performance tests as part of CI.

    However, we use a custom test runner to suit our needs. Running tests step runs a batch file which calls our custom test runner.

    Can dotTrace work with customer test runner?

  5. JA says:

    Feature request: is it possible to store the results per-agent to reduce the need to limit the build configuration to a single agent? Or otherwise to limit the step to run (or report failure) on only one build agent in the pool? Of course, I can set up a separate build configuration purely for performance tests, but I’d rather get the feedback during the 1st stage of CI.

  6. Shagin Sasi says:

    Hi
    We integrated Performance profiling in Continuous Integration using dotTrace plugin to TeamCity. But we facing issue Opening Snapshot in dotTrace Perfromance Viewer as it showing “Cannot read snapshot file”. We are using JetBeans dotTrace Performance Viewer 2016.3.2 version and the plugin from http://teamcity.jetbrains.com/httpAuth/app/rest/builds/buildType:TeamCityPluginsByJetBrains_DotTrace_Build,pinned:true,status:SUCCESS,branch:master/artifacts/content/dotTrace.zip location.

  7. Roger says:

    Hi

    Two feature requests…

    * Would like ONE snapshot file instead. At least have them zipped. This way it’s simpler for TC user to download/run it in their local dottrace.
    * For some reason it seems mandatory to set at least one threshold value. We don’t want that, we simply want a snapshot produced.

    Thanks!
    /Roger

Leave a Reply

Your email address will not be published. Required fields are marked *