dotTrace comes to BenchmarkDotNet

Khalid Abuhakmeh

BenchmarkDotNet is the premiere benchmarking suite for developers building .NET applications. With BenchmarkDotNet, you can run benchmarks to track performance and share reproducible experiments with team members. This project has helped 16,300+ projects, including the .NET team, iterate on optimizations and quickly flag regressions.

One of BenchmarkDotNet’s greatest strengths is its Diagnosers, which help surface specific aspects of a benchmark run, including GC pressure, Memory allocations, JIT Inlining Events, and more, and with the complexity of a running application, the more diagnosers, the better.

Wouldn’t it be great if there was a diagnoser that lets you dive deeper into a benchmark’s performance snapshot?

Today we’re happy to announce that our JetBrains team, including BenchmarkDotNet project lead Andrey Akinshin, are bringing a new DotTraceDiagnoser to performance-minded JetBrains users everywhere.

This post explores applying the new DotTraceDiagnoser to a benchmark suite and how to view the profiling snapshot in JetBrains tools like JetBrains Rider and dotTrace.

Getting Started with BenchmarkDotNet and dotTrace

Developers have benchmarks because they’re either curious about their application’s performance profile or looking for areas that could use improvement. BenchmarkDotNet can help you better understand your application’s performance profile with a few baked-in mechanisms.

While not necessary for a core BenchmarkDotNet experience, Diagnosers can increase your ability to capture detailed information about a specific benchmark run, including memory, GC calls, and ETW calls. As you’d expect, diagnoser output depends on its implementation but typically is in a format that can be challenging to read. A more user-friendly diagnoser matched with a thoughtful exporter could help users better understand and manage their performance-critical codebases, and dotTrace and the new diagnoser can do just that.

dotTrace, our performance profiler, can help you uncover the performance mysteries of your codebase in what we call an investigation. A profiling investigation can make spotting issues more straightforward with the ability to filter the call tree, highlight hotspots, exclude system calls, and view a flame graph. These features can be invaluable when you’re working to squeeze out every last bit of application performance.

Let’s see how you can add a DotTraceDiagnoser to your BenchmarkDotNet benchmark suite and see the inner workings of your runs.

dotTrace Diagnoser and Benchmarks

We recommend you read the BenchmarkDotNet documentation, as it’s highly detailed about all aspects of the library. For this article, we’ll run a dotTrace profiling session on a Fibonacci benchmark run.

Before looking at the code, you will need two packages from version v0.13.6+ of BenchmarkDotNet, available now on NuGet. To start, run the following commands on a new or existing .NET console application.

dotnet add package BenchmarkDotNet
dotnet add package BenchmarkDotNet.Diagnostics.dotTrace

You’ll also need to install the SimpleGames.GameOfLife NuGet package, which provides an implementation of Conway’s Game of Life, an intensive iterative algorithm. When the installation is complete, you can replace the contents of Program.cs with the following code.

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Diagnostics.dotTrace;
using BenchmarkDotNet.Running;
using SimpleGames.GameOfLife;

BenchmarkRunner.Run<IntroDotTraceDiagnoser>();

[DotTraceDiagnoser]
[SimpleJob, InProcess]
public class IntroDotTraceDiagnoser
{
    private readonly GameOfLifeGeneration game = 
        GameOfLifeGeneration.Parse(
        @"...........
          ....O......
          ...O.......
          ...OOO.....
          ...........
          ........... 
          .O.........
          O.O........
          .O.........
        ");
    
    [Benchmark]
    public void LifeFindsAWay() =>
        this.game.Next(200);
}

You’ll note a DotTraceDiagnoser attribute on our `IntroDotTraceDiagnoser` benchmark class. The attribute tells BenchmarkDotNet to perform an additional run to produce a profiling snapshot during the run. The final results you typically see in the output will not be affected by the profiling run. Additionally, BenchmarkDotNet provides two mechanisms for profiling our benchmarks.

• ExternalProcess: This uses dotTrace’s command-line profiler. These tools will be downloaded and installed for you automatically.
• InProcess: This uses JetBrains.Profiler.SelfApi to download the latest dotTrace tooling.

When using the SimpleJobAttribute, BenchmarkDotNet defaults to using the external process while adding the InProcessAttribute will queue the profiling job within the current process. In specific scenarios, you may be unable to run and control an external process, so adding the additional attribute is a known workaround.

From a developer’s perspective, they produce the same expected final results, a dotTrace profiling session snapshot. Choose whichever works best for you and your development environment.

Running the benchmark, you will see the following output.

// * Diagnostic Output - DotTrace *
The following dotTrace snapshots were generated:
* /Users/me/RiderProjects/BenchmarkDotTraceSample/BenchmarkDotTraceSample/bin/Release/net7.0/BenchmarkDotNet.Artifacts/snapshots/IntroDotTraceDiagnoser.LifeFindsAWay-.NET 7.0-20230609-135855.dtp
* /Users/me/RiderProjects/BenchmarkDotTraceSample/BenchmarkDotTraceSample/bin/Release/net7.0/BenchmarkDotNet.Artifacts/snapshots/IntroDotTraceDiagnoser.LifeFindsAWay-InProcessEmitToolchain-20230609-135905.dtp

As the output shows, BenchmarkDotNet writes all snapshots to a BenchmarkDotNet.Artifacts folder under a target framework’s bin folder. In the case of this sample, we ran both the InProcess and ExternalProcess profilers to show that they both work as expected, which explains why we see two files.

The next step is to view the snapshots and start your investigation.

Viewing dotTrace Snapshots

You will need a tool to open the .dtp format to view snapshots. Your options include JetBrains Rider and dotTrace. All are available in the dotUltimate license and installed through the JetBrains Toolbox app.

In JetBrains Rider, click the dotTrace Profiler tool window and then the folder icon on the left-hand side to open an existing snapshot.

Once you’ve opened the snapshot, you can view the session results.

The results show that the CoreNextGeneration method is a hotspot, a message the flame graph reinforces. Not surprising seeing it’s the only method in the demo.

You can also open snapshots with dotTrace. The portability of snapshots makes it possible to generate benchmarks in a CI/CD environment and investigate results locally. The following shows the same snapshot in the macOS version of dotTrace (currently in Beta). The Windows version of the tool also opens the snapshot file.

Regardless of operating system and development tools, you can take full advantage of dotTrace profiling in your BenchmarkDotNet test suite.

BenchmarkDotnet and dotTrace Caveats

While the DotTraceDiagnoser is a fantastic addition to your benchmark suites, there are notable considerations you should be aware of. 

When writing this post, sampling is the only supported profiling approach. Sampling accurately measures call times, but you may need some information only found in Timeline, Tracing, and Line-by-line sessions. If you require that additional information, you should perform further profiling with dotTrace after narrowing investigations down to known hotspots.

The `DotTraceDiagnoser` does not currently support Mono. If you’re doing benchmark runs on the Mono runtime, this approach to benchmarking will not work.

Note that the BenchmarkDotNet integration with dotTrace still has the same limitations as our dotTrace command-line tools. Please consult our dotTrace command-line tools documentation to determine if this works for your use case.

Conclusion

An essential part of optimizing applications is measuring performance and interpreting those results. BenchmarkDotNet helps developers uncover performance bottlenecks through diagnosers. One of the more powerful diagnosers, dotTrace can help developers find more opportunities to optimize performance and ship great user experiences.

We want you to try this newest integration between BenchmarkDotNet and dotTrace profiling tools, and let us know what you think.

As always, we look forward to your feedback and comments.

Subscribe to a monthly digest curated from the .NET Tools blog:

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!