Better Ways to Test with doctest – the Fastest C++ Unit Testing Framework

Which C++ unit testing framework do you use? In this guest blog post, Viktor Kirilov shares how Doctest, a new C++ testing framework he contributes to, is better than others.

Viktor Kirilov Viktor Kirilov

GitHub
@KirilovVik
Viktor is a huge fan of the Nim programming language and recently worked on adding support for hot code reloading at runtime in the compiler. Obsessed about developer productivity; most of his public talks have been about optimizing the compilation time of C++ programs and reloading code at runtime. He is the author of doctest – the fastest C++ unit testing framework.

doctest is a relatively new C++ testing framework but is by far the fastest both in terms of compile times (by orders of magnitude) and runtime compared to other feature-rich alternatives. It was released in 2016 and has been picking up in popularity ever since.

A complete example with a self-registering test that compiles to an executable looks like this:

There is no need to link to anything – the library is just a single header which depends only on the C++ standard library. The output from that program is the following:

A list of some of the important features can be summarized as follows:

What makes doctest interesting

So far doctest sounds like just another framework with some set of features. What truly sets it apart is the ability to use it alongside your production code. This might seem strange at first, but writing your tests right next to the code they are testing is an actual pattern in other languages such as Rust, D, Nim, Python, etc. – their unit testing modules let you do exactly that.

But why is doctest the most suitable C++ framework for this? A few key reasons:

  • Ultra light – less than 20 ms of compile time overhead for including the header in a source file.
  • The fastest possible assertion macros – 50,000 asserts can compile for under 20 seconds (even under 10 sec).
  • Offers a way to remove everything testing-related from the binary with the DOCTEST_CONFIG_DISABLE identifier (for the final release builds).
  • Doesn’t produce any warnings even on the most aggressive levels for MSVC / GCC / Clang.
  • Very portable and well-tested C++11 – per commit tested on CI with over 180 different builds with different compilers and configurations (gcc 4.8-9.1 / clang 3.5-8.0 / MSVC 2015-2019, debug / release, x86/x64, linux / windows / osx, valgrind, sanitizers, static analysis…).

The idea is that you shouldn’t even notice if there are tests in the production code – the compile time penalty is negligible and there aren’t any traces of the testing framework (no warnings, no namespace pollution, and macros and command line options can be prefixed). The framework can still be used like any other even if the idea of writing tests in the production code doesn’t appeal to you. Yet, this is the biggest power of the framework and nothing else comes even close to being so practical in achieving this. Think of the improved workflow:

  • The barrier for writing tests becomes much lower – you will not have to:
    1. Make a separate source file.
    2. Include a bunch of headers in it.
    3. Add it to the build system.
    4. Add it to source control.
    5. Wait for excessive compile + link times (because your heavy headers would need to be parsed an extra time and the static libraries you link against are a few hundred megabytes).
  • You can just write the tests for a class or a piece of functionality at the bottom of its source file (or even header file)!
  • Tests in the production code can be thought of as inline documentation, showing how an API is used (correctness enforced by the compiler – always up-to-date).
  • Testing internals that are not exposed through the public API and headers becomes easier.

Integration within programs

Having tests next to your production code requires a few things:

  • Everything testing-related should be optionally removable from builds.
  • Code and tests should be executable in 3 different scenarios: only the tests, only the program, and both.
  • Programs consisting of an executable + multiple shared objects (.dll/.so/.dylib) should have a single test registry.

The effect of the DOCTEST_CONFIG_DISABLE identifier when defined globally in the entire project is that the TEST_CASE() macro becomes the following:

There is no instantiation of the anonymous template and there is no test self-registration – the test code will not be present in the final binaries even in Debug. The other effects of this identifier are that asserts within the test case body are turned into no-ops, so even less code is parsed/compiled within these uninstantiated templates, and the test runner is almost entirely removed. Using this identifier is equivalent to not having written any tests – they simply no longer exist.

Here is an example main() function showing how to foster the 3 execution scenarios when tests are present (also showing how defaults and overrides can be set for command line options):

With this setup, the following 3 scenarios are possible:

  • Running only the tests (with the –exit option or just doing a query like listing all test cases).
  • Running only the user code (with the –no-run option to the test runner).
  • Running both the tests and the user code.

In the case of programs comprised of multiple binaries (shared objects), the DOCTEST_CONFIG_IMPLEMENTATION_IN_DLL identifier can be used – then only a single binary should provide the test runner implementation. Even plugins that are loaded by the program after it has started will properly register their tests into the registry, which should be separated into a common shared library to which every other binary links against (see this example).

Going a step further – using doctest as a general-purpose assert library

Perhaps you use some custom assert for checking preconditions in the actual code. That assert won’t play nicely within a testing context (failures won’t be handled uniformly). Wouldn’t it be nice if we could just use doctest asserts instead? Turns out that’s possible – this way a project could have a unified way of asserting invariants both in production code and in test scenarios – with the use of a single set of macros and a single point of configuration!

All the user has to do is set a doctest::Context object somewhere as the default for asserts outside of a testing context. Asserts will call std::abort on failure, but this behavior can be overridden by setting an assert handler – with a call to setAssertHandler() on the context. The handler is a function with the following signature: “void handler(const doctest::AssertData&)” and everything important for the assert can be extracted through the AssertData input. It can choose to abort, throw, or even just to log an entry for the failure somewhere – the choice is yours! An example of what that would look like can be seen here. Thankfully, doctest is thread-safe – there is nothing stopping us from using the same set of asserts in any context!

This would be best combined with the use of the binary asserts which are faster for compilation than the normal expression-decomposing ones (less template instantiations). And why not use the DOCTEST_CONFIG_SUPER_FAST_ASSERTS identifier to reach the best possible compile time, turning each assert into a single function call?

Conclusion

Testing is a fundamental aspect of software engineering and the stakes are getting only higher. The world runs entirely on software and the responsibility is placed upon us to develop and enforce standards and procedures in the fastest changing and least mature industry. Using better tools that remove friction in the development process is the best approach towards a more robust and secure future – human nature should never be left out of the equation.

doctest stands out with its ability to write tests in a new and easier way, unlocking the potential for more thorough, up-to-date, and uniform testing. Locality is king not only in CPU caches. There is quite a lot of work left which can be seen in the roadmap – exciting times are ahead of us! If you are curious about the implementation details of the framework, make sure to check out the CppCon presentation!

Doctest support in ReSharper C++

Starting with v2019.1, ReSharper C++ supports Doctest, in addition to Google Test, Boost.Test, and Catch.

When you have doctest.h header included, ReSharper C++ discovers Doctest test cases and suites and adds a corresponding indicator next to each one in the editor. With it you can Run or Debug the test or the whole test suite:
doctest_run_icon

You can even see the status from the last execution (a green mark is added for tests that passed successfully, and a red one for failed tests):
doctest_suite

The Unit Test Sessions window will be opened for you to explore the execution progress, results, and output of the tests you run or debug:
doctest_unit_test_session
On the status bar, you will see the total number of tests in the session as well as the number of tests in different states, such as passed, failed, or ignored, for example.

To explore all the tests in the entire solution, use the Unit Test Explorer window (Ctrl+Alt+T):
doctest_explorer

If you want to learn more about ReSharper C++ unit testing support, please read the official documentation.

We encourage you to try Doctest along with ReSharper C++ support for it and share your feedback and ideas here in comments!

This entry was posted in Uncategorized and tagged , , , . Bookmark the permalink.

9 Responses to Better Ways to Test with doctest – the Fastest C++ Unit Testing Framework

  1. George says:

    The reason I did not use any of the ready test frameworks was that the tests were written on separate files than the class they are testing.
    I made my own minimal test functionality
    https://github.com/starmessage/cpcc/blob/master/cpcc_SelfTest.h
    to write tests like
    SELFTEST_BEGIN(cpccFileSystemMini_SelfTest)
    cpccFileSystemMini::selfTest();
    SELFTEST_END
    where every class has a static function selfTest() that makes all the needed tests.

    It’s good to find your doctest library.
    I will try it in my SoftMeter application analytics library. But, is it possible to have some tests only for the debug builds and some tests to be kept also on the release builds?

    • Viktor Kirilov says:

      Hello George,

      As mentioned in the article you could remove all tests from the final release build with the help of DOCTEST_CONFIG_DISABLE. If however you would like to remove only some of them you could instead make a proxy header file which includes doctest but defines a few macros – like TEST_CASE_ALWAYS_PRESENT and TEST_CASE_NORMAL and based on some preprocessor identifier of your choice to either have those macros just forward to TEST_CASE from doctest or do the uninstantiated template trick as shown in the article. You could also give the tests names based on some scheme and even if you compile all tests you could choose to execute only a subset of them – with the help of the command line filtering –test-case= option.

      Let me know if this helps!

  2. d21240 says:

    We have been using the GoogleTest framework, perhaps you can explain the differences? Kind regards, David

    • Viktor Kirilov says:

      Hi David,

      Here are a couple of differences:
      – the main one is that only doctest from the C++ frameworks is usable next to your production code – that is what this article is about (speed of compilation, ability to remove the tests from the binary, ability to execute tests/code/both, ability to have tests in multiple shared objects and still a single registry for all of them)
      – doctest is a single header – google test has to be built as a separate static library and linked against.
      – doctest has the concept of Subcases (see the link in the article or the tutorial) which is a much cleaner way to share setup and teardown code between tests compared to fixtures and class inheritance – google test is quite verbose!
      – doctest compiles faster and probably runs faster (although the runtime becomes an issue only when you have more than a couple hundred thousand asserts or even in the millions)
      – doctest asserts are thread-safe even on Windows (google test uses pthreads so thread-safe asserts are present only on UNIX)
      – doctest overall has a simpler API

      but there are also some things in which doctest is lacking:
      – value-parameterized tests
      – death tests (where you check if calling a certain function doesn’t simply throw but if it crashes the process)
      – doctest has some integration with mocking libraries but google test works perfectly with google mock (although doctest should in theory work with it as well)

      The areas where doctest is behind are planned for improvement in the future. There are many other smaller differences – it would be impractical to cover them all.

      Checkout the tutorial and the reference documentation for more information :)
      https://github.com/onqtam/doctest/blob/master/doc/markdown/tutorial.md
      https://github.com/onqtam/doctest/blob/master/doc/markdown/readme.md

  3. Tom says:

    I’m eagerly awaiting doctest support in CLion. Please tell me this is planned?

  4. Alex says:

    The framework looks great!
    I would prefer it to be named “doctestcpp” or something, so when I try to find more info about it I don’t get lots of Python doctest search results :)

    We’ll give it a try at work for sure.

  5. Dinesh says:

    Is there a way that i can call the TEST_CASE when i need to rather than at the launch of the executable

Leave a Reply

Your email address will not be published. Required fields are marked *