Better Ways to Test with doctest – the Fastest C++ Unit Testing Framework

Anastasia Kazakova

Which C++ unit testing framework do you use? In this guest blog post, Viktor Kirilov shares how Doctest, a new C++ testing framework he contributes to, is better than others.

Viktor Kirilov GitHub @KirilovVik Viktor is a huge fan of the Nim programming language and recently worked on adding support for hot code reloading at runtime in the compiler. Obsessed about developer productivity; most of his public talks have been about optimizing the compilation time of C++ programs and reloading code at runtime. He is the author of doctest – the fastest C++ unit testing framework.

doctest is a relatively new C++ testing framework but is by far the fastest both in terms of compile times (by orders of magnitude) and runtime compared to other feature-rich alternatives. It was released in 2016 and has been picking up in popularity ever since.

A complete example with a self-registering test that compiles to an executable looks like this:

#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"

int fact(int n) {
    return n <= 1 ? n : fact(n - 1) * n;
}

TEST_CASE("testing the factorial function") {
    CHECK(fact(0) == 1); // should fail
    CHECK(fact(1) == 1);
    CHECK(fact(2) == 2);
    CHECK(fact(3) == 6);
    CHECK(fact(10) == 3628800);
}

There is no need to link to anything – the library is just a single header which depends only on the C++ standard library. The output from that program is the following:

[doctest] doctest version is "2.3.3"
[doctest] run with "--help" for options
===============================================================================
hello_world.cpp:8:
TEST CASE:  testing the factorial function

hello_world.cpp:9: ERROR: CHECK( fact(0) == 1 ) is NOT correct!
  values: CHECK( 0 == 1 )

===============================================================================
[doctest] test cases:      1 |      0 passed |      1 failed |      0 skipped
[doctest] assertions:      5 |      4 passed |      1 failed |
[doctest] Status: FAILURE!

A list of some of the important features can be summarized as follows:

• Expression decomposition – use standard comparison operators in asserts instead of having to explicitly say if the assert is for equality, less than, etc.
• Thread-safe asserts which can be used in a multi-threaded context.
• Can be used without exceptions and RTTI.
• Subcases – an intuitive way to share common setup and teardown code for test cases (inspired by sections in Catch2).
• Crash handling, logging, an extensible reporter system (xml, custom), templated test cases, test suites, decorators, a rich command line and many more.

What makes doctest interesting

So far doctest sounds like just another framework with some set of features. What truly sets it apart is the ability to use it alongside your production code. This might seem strange at first, but writing your tests right next to the code they are testing is an actual pattern in other languages such as Rust, D, Nim, Python, etc. – their unit testing modules let you do exactly that.

But why is doctest the most suitable C++ framework for this? A few key reasons:

• Ultra light – less than 20 ms of compile time overhead for including the header in a source file.
• The fastest possible assertion macros – 50,000 asserts can compile for under 20 seconds (even under 10 sec).
• Offers a way to remove everything testing-related from the binary with the DOCTEST_CONFIG_DISABLE identifier (for the final release builds).
• Doesn’t produce any warnings even on the most aggressive levels for MSVC / GCC / Clang.
• Very portable and well-tested C++11 – per commit tested on CI with over 180 different builds with different compilers and configurations (gcc 4.8-9.1 / clang 3.5-8.0 / MSVC 2015-2019, debug / release, x86/x64, linux / windows / osx, valgrind, sanitizers, static analysis…).

The idea is that you shouldn’t even notice if there are tests in the production code – the compile time penalty is negligible and there aren’t any traces of the testing framework (no warnings, no namespace pollution, and macros and command line options can be prefixed). The framework can still be used like any other even if the idea of writing tests in the production code doesn’t appeal to you. Yet, this is the biggest power of the framework and nothing else comes even close to being so practical in achieving this. Think of the improved workflow:

• The barrier for writing tests becomes much lower – you will not have to:
  1. Make a separate source file.
  2. Include a bunch of headers in it.
  3. Add it to the build system.
  4. Add it to source control.
  5. Wait for excessive compile + link times (because your heavy headers would need to be parsed an extra time and the static libraries you link against are a few hundred megabytes).
• You can just write the tests for a class or a piece of functionality at the bottom of its source file (or even header file)!
• Tests in the production code can be thought of as inline documentation, showing how an API is used (correctness enforced by the compiler – always up-to-date).
• Testing internals that are not exposed through the public API and headers becomes easier.

Integration within programs

Having tests next to your production code requires a few things:

• Everything testing-related should be optionally removable from builds.
• Code and tests should be executable in 3 different scenarios: only the tests, only the program, and both.
• Programs consisting of an executable + multiple shared objects (.dll/.so/.dylib) should have a single test registry.

The effect of the DOCTEST_CONFIG_DISABLE identifier when defined globally in the entire project is that the TEST_CASE() macro becomes the following:

#define TEST_CASE(name) \
    template <typename T> \
    static inline void ANONYMOUS(ANON_FUNC_)()

There is no instantiation of the anonymous template and there is no test self-registration – the test code will not be present in the final binaries even in Debug. The other effects of this identifier are that asserts within the test case body are turned into no-ops, so even less code is parsed/compiled within these uninstantiated templates, and the test runner is almost entirely removed. Using this identifier is equivalent to not having written any tests – they simply no longer exist.

Here is an example main() function showing how to foster the 3 execution scenarios when tests are present (also showing how defaults and overrides can be set for command line options):

#define DOCTEST_CONFIG_IMPLEMENT
#include "doctest.h"

int main(int argc, char** argv) {
    doctest::Context ctx;

    ctx.setOption("abort-after", 5);  // default - stop after 5 failed asserts

    ctx.applyCommandLine(argc, argv); // apply command line - argc / argv

    ctx.setOption("no-breaks", true); // override - don't break in the debugger

    int res = ctx.run();              // run test cases unless with --no-run

    if(ctx.shouldExit())              // query flags (and --exit) rely on this
        return res;                   // propagate the result of the tests

    // your actual program execution goes here - only if we haven't exited

    return res; // + your_program_res
}

With this setup, the following 3 scenarios are possible:

• Running only the tests (with the –exit option or just doing a query like listing all test cases).
• Running only the user code (with the –no-run option to the test runner).
• Running both the tests and the user code.

In the case of programs comprised of multiple binaries (shared objects), the DOCTEST_CONFIG_IMPLEMENTATION_IN_DLL identifier can be used – then only a single binary should provide the test runner implementation. Even plugins that are loaded by the program after it has started will properly register their tests into the registry, which should be separated into a common shared library to which every other binary links against (see this example).

Going a step further – using doctest as a general-purpose assert library

Perhaps you use some custom assert for checking preconditions in the actual code. That assert won’t play nicely within a testing context (failures won’t be handled uniformly). Wouldn’t it be nice if we could just use doctest asserts instead? Turns out that’s possible – this way a project could have a unified way of asserting invariants both in production code and in test scenarios – with the use of a single set of macros and a single point of configuration!

All the user has to do is set a doctest::Context object somewhere as the default for asserts outside of a testing context. Asserts will call std::abort on failure, but this behavior can be overridden by setting an assert handler – with a call to setAssertHandler() on the context. The handler is a function with the following signature: “void handler(const doctest::AssertData&)” and everything important for the assert can be extracted through the AssertData input. It can choose to abort, throw, or even just to log an entry for the failure somewhere – the choice is yours! An example of what that would look like can be seen here. Thankfully, doctest is thread-safe – there is nothing stopping us from using the same set of asserts in any context!

This would be best combined with the use of the binary asserts which are faster for compilation than the normal expression-decomposing ones (less template instantiations). And why not use the DOCTEST_CONFIG_SUPER_FAST_ASSERTS identifier to reach the best possible compile time, turning each assert into a single function call?

Conclusion

Testing is a fundamental aspect of software engineering and the stakes are getting only higher. The world runs entirely on software and the responsibility is placed upon us to develop and enforce standards and procedures in the fastest changing and least mature industry. Using better tools that remove friction in the development process is the best approach towards a more robust and secure future – human nature should never be left out of the equation.

doctest stands out with its ability to write tests in a new and easier way, unlocking the potential for more thorough, up-to-date, and uniform testing. Locality is king not only in CPU caches. There is quite a lot of work left which can be seen in the roadmap – exciting times are ahead of us! If you are curious about the implementation details of the framework, make sure to check out the CppCon presentation!

Doctest support in ReSharper C++

Starting with v2019.1, ReSharper C++ supports Doctest, in addition to Google Test, Boost.Test, and Catch.

Update: Since v2020.2 Doctest is also supported in CLion.

When you have doctest.h header included, ReSharper C++ discovers Doctest test cases and suites and adds a corresponding indicator next to each one in the editor. With it you can Run or Debug the test or the whole test suite:

You can even see the status from the last execution (a green mark is added for tests that passed successfully, and a red one for failed tests):

The Unit Test Sessions window will be opened for you to explore the execution progress, results, and output of the tests you run or debug:

On the status bar, you will see the total number of tests in the session as well as the number of tests in different states, such as passed, failed, or ignored, for example.

To explore all the tests in the entire solution, use the Unit Test Explorer window (Ctrl+Alt+T):

If you want to learn more about ReSharper C++ unit testing support, please read the official documentation.

We encourage you to try Doctest along with ReSharper C++ support for it and share your feedback and ideas here in comments!

Subscribe to ReSharper C++ Blog updates

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!