CLion 2019.3 EAP: Code Coverage and CMake Defaults

Posted on by Anastasia Kazakova

Hi,

A new CLion 2019.3 EAP (build 193.4932.12) is now available! Get it from our website, via the Toolbox App, or as a snap package (if you are using Ubuntu). A patch-update for anyone using the previous EAP build will be available shortly.

DOWNLOAD CLION 2019.3 EAP

Key highlights:

Code Coverage

Code coverage in CLion relies on llvm-cov/gcov integration to perform and show you statements coverage measuring for your code. In general, statements coverage answers the question, "Was this statement executed during the configuration run?". The configuration can be either a unit tests run or a regular configuration run.

How to launch a configuration with coverage?

To get coverage measurement, you have to pass special coverage compile options. There are a few options depending on what compiler you use and what kind of results you expect:

  • -fprofile-arcs -ftest-coverage or an equivalent --coverage – for GCC (uses gcov) and Clang (uses llvm-cov gcov, gcov-style coverage, which means it will compatible with the gcov tool from version 4.2 of GCC)
  • -fprofile-instr-generate -fcoverage-mapping – for Clang (uses llvm-profdata merge / llvm-cov export, a clang’s instrumentation based profiling).

For example, if you use both GCC and Clang in your project, you may have something like this in your CMakeLists.txt:
coverage_cmake

Now you can run it using the run icon in the left gutter:
coverage_run

Or you can click the “Run … with Coverage” button coverage_icon in the Navigation Bar.

How to read the results

Coverage measurements are located in the Coverage tool window, which shows results per folder (% of files and % of lines covered) and per file (% of lines covered):
coverage_results

When rerunning the coverage analysis, CLion will ask you how you’d like the results presented:

  • Add to active suites – to enrich the current results.
  • Replace active suites – to start from scratch.
  • Do not apply collected coverage data – to simply ignore the current results.

Another way of exploring the results is to check the color indication in the left gutter in the editor.
coverage_indication_gutter

Click the gutter area to see how many times the line was hit.

Configuration and known issue

Coverage settings, including custom paths to gcov / llvm-cov / llvm-profdata tools and more, can be found in Settings/Preferences | Build, Execution, Deployment | Coverage:
coverage_settings

A few important known issues are worth mentioning:

  • Code coverage doesn’t work with Ninja generator (CPP-17864). We’ll do our best to address this issue before the release.
  • Code Coverage is not supported for the remote toolchain (CPP-17709).
  • Generate Coverage Report – an action you may be familiar with if you use IntelliJ IDEA – is not available yet (CPP-17710).

Other bugs and requests are linked to the parent ticket, CPP-9678. Please try the EAP build with code coverage support and share your feedback with us!

CMake defaults

If you have to work with a lot of new projects in CLion, you may want to simplify the process of configuring CMake Profiles, for example by configuring a common pattern for the CMake generation folder.

Starting with this EAP, you can configure one or several default CMake Profiles which will be used for all your new CLion projects! Use File | Other Settings | Settings/Preferences for New Projects…:
cmake_settings

You can set the default:

  • Generator used in CMake
  • Options passed to CMake
  • Toolchain
  • Build type
  • Environment
  • Generation path
  • Build options

When a new project is started, these default CMake Profiles will be used. Note that currently, this doesn’t apply when File | New CMake Project from Sources is used (CPP-17686).

That’s it for today! The full release notes are available here.

DOWNLOAD CLION 2019.3 EAP

Your CLion Team
JetBrains
The Drive to Develop

Comments below can no longer be edited.

46 Responses to CLion 2019.3 EAP: Code Coverage and CMake Defaults

  1. Martin Pecka says:

    October 30, 2019

    Nice, the default CMake profiles might come handy.

    When I set an absolute generation path, can I use some variables in the input? If I for example wanted to put all CLion build directories under /data/clion/{project-name}/cmake-build-debug …

    • Anastasia Kazakova says:

      October 31, 2019

      It can be an absolute path or a relative path to project root.

      • Martin Pecka says:

        November 1, 2019

        And do you plan adding support for variables in this path?

  2. mbalabin says:

    October 31, 2019

    Reading a local .lldbinit feature is a nice little feature! But it seems that it has not been released yet. The corresponding ticket (https://youtrack.jetbrains.com/issue/CPP-17608) is still in progress.

    • Eldar Abusalimov says:

      October 31, 2019

      I’m afraid I owe an apology on this one. The feature was not included into the released build indeed. I got switched to some other tasks, and forgot to put the necessary changes into the release branch. Sorry for the confusion.

  3. Henry says:

    October 31, 2019

    oh my god! Coverage!!! This is awesome!

  4. Jonathan Ross says:

    October 31, 2019

    Like Henry, I’m happy to see code coverage added to CLion! (I have unfortunately not yet gotten it to work: my coverage results are empty.)

    It seems error-prone to have to add coverage compilation flags to my CMake projects by hand to get it to work. Are there any plans to automate this/add the flags behind the scenes?

    • Anastasia Kazakova says:

      November 1, 2019

      Hi,

      Feel free to issue a bug if case coverage still fails, We’ll request some additional logs and info and will try to help.

      As for the automation, we are still unsure here. In general, we prefer not to interfere in the CMake process from the IDE’s side, which means not adding any flags by the IDE itself. But we understand the current workflow is not very convenient and user-friendly. So we are thinking about the solution.

      • Jonathan Ross says:

        November 7, 2019

        I managed to get the code coverage working: it turned out that gcov was aliased to llvm-cov on my machine (Mac). When I configured the path explicitly it worked great. My one remaining gripe would be that the coverage analysis failed silently when I had it configured incorrectly.

        Regarding automation / not interfering with the CMake process: one solution that springs to mind, would be for you to intercept at the toolchain level: wrap the compiler with a script that appends the relevant flags.

        • Anastasia Kazakova says:

          November 8, 2019

          Good to know it works! We’ll see if we can provide better logging.

          Regarding the wrapping, it’s indeed one of the solutions. But also has its drawbacks.

  5. David Zemon says:

    November 2, 2019

    Finally got around to trying this today. Very excited to see the familiar coverage GUI back in my toolbox, after moving from Java to C/C++ a couple years ago.

    One big disappointment though: it doesn’t seem possible to tell CLion that multiple coverage reports need to be combined together. This makes it far more difficult to get the whole-project-picture of code coverage.

    • Anastasia Kazakova says:

      November 4, 2019

      What do you mean by combining the reports? When you rerun the coverage, CLion asks if you’s like to add to the current report, drop results or start a new one. Do you mean smth different here?

      • Helge Penne says:

        November 5, 2019

        I do not get any questions either about add/drop/new when I rerun the coverage. Known bug?

  6. Andrew Smith says:

    November 4, 2019

    It looks like the new “Remote GDB Server” still requires that one have a project that builds locally, and runs the GDB client locally. My project cannot build with CLion locally or remotely; I build it outside of CLion. Is there a way to remotely debug over SSH without a local build (similar to how it works with a remote toolchain, but just running a remote process built through an external build system)?

    • Anastasia Kazakova says:

      November 5, 2019

      For such cases we have GDB Remote Debug configuration: https://www.jetbrains.com/help/clion/2019.3/remote-debug.html
      You’ll have to manually start the gdbserver, provide CLion with the path mappings and symbols for the debug (this is necessary as you build outside of CLion).

      • Andrew Smith says:

        November 6, 2019

        The problem with that is that this would require transferring symbols every time I debug. My symbols are over 1gb, which is prohibitive to transfer every time. Your SSH debugging with a remote tool chain (using gdb over SSH) works perfectly, except that CLion only supports that with projects that build remotely in CLion.

        It seems like you have all of the hard stuff done for GDB over SSH, and the only thing missing is a simple UI entry point to support it for projects that build outside of CLion. Or is there another way to do this that I’m missing?

        • Anastasia Kazakova says:

          November 6, 2019

          Do you still build locally? Or is it a 3-rd machine?

          • Andrew Smith says:

            November 6, 2019

            I build on a remote machine (the same machine I want to debug on). The code won’t build locally, both because of a different build system and different local OS. I have confirmed that SSH debugging works with a CLion remote tool chain project (where CLion builds the project remotely); there just doesn’t seem to be a way to do it in the UI outside of a remote tool chain project (which requires CLion to do the build).

            • Anastasia Kazakova says:

              November 7, 2019

              Ok, but not sure I get the problem why you can’t use the Full Remote mode, which builds on a remote machine. Indeed it’s done from the CLion. Why it doesn’t work for you? Do you have some custom build different from CMake?

          • Andrew Smith says:

            November 8, 2019

            Yes, we use a completely different build system (with distributed caching of intermediate artifacts/etc).

            • Anastasia Kazakova says:

              November 11, 2019

              Ok, but CLion has to get the build symbols for the debug somehow. Either it gets them when building on its own, or you upload it to the host where CLion is on your own. You can probably configure some rsync or similar to do that for you automatically.

          • Andrew Smith says:

            December 2, 2019

            In your “full remote mode”, CLion does not need the debug symbols. Instead, it communicates with GDB client on the remote server over ssh, using stdin/stdout to drive GDB remotely. That avoids a multi-gigabyte transfer of symbols every time one presses play.

            Since this already works in “full remote mode” (with no symbols), would it be possible to enable this scenario outside of full remote mode? It seems like all that would be needed is a UI entrypoint, since you already did the work to drive GDB via stdin/stdout over ssh.

          • Vasily Romanikhin says:

            December 4, 2019

            @Andrew Smith Thanks.
            That’s an interesting case, unfortunnately at the moment there is no official way to do that. I’ve created a proper feature-request, feel free to comment and vote (https://youtrack.jetbrains.com/issue/CPP-18311).

  7. Helge Penne says:

    November 5, 2019

    Integrated coverage is really nice. I’ve given it a test run and here is what I’ve found so far:
    – It works really well for small tests executables.
    – When I ran it on a test executable with a few thousand tests in it, CLion was stuck in processing for at least 15 minutes and eventually failed on an out-of-memory error.
    – Coverage processing seems single treaded, which is a pity when it takes quite some time to process large sets of tests. It seems you should work more on speed and memory usage. The current implementation will not be good enough for large projects.
    – I do not get the question about add/drop/new etc. that is mentioned on the blog when I re-run tests in coverage mode. This seems like a bug.

    The slow processing and out-of-memory failure is sad. This was with clang. Is there any hope of better results with a gcc toolchain? Can you improve on this?

    • Anastasia Kazakova says:

      November 5, 2019

      Can you please submit a bug report? We’ll probably need some more information to investigate the issues. Thanks.

    • vplyashkun says:

      November 5, 2019

      Hi Helge Penne!
      Thanks for the report.
      Am i right that you use clang with -fprofile-arcs -ftest-coverage or an equivalent --coverage compiler option(s)?

      As for out of memory exception, please try enabling memory indicator (Settings | Appearance & Behavior | Appearance | Show memory indicator) and keep an eye on it for some time. If it is close to the limit at the time when you are experiencing the problems, then try increasing the Xmx JVM. Does that help?

      If it still fails, please capture memory snapshot (https://intellij-support.jetbrains.com/hc/en-us/articles/207241235-Reporting-performance-problems) to help us with investigation.

      • Helge Penne says:

        November 5, 2019

        I’ve created a bug report for this:
        https://youtrack.jetbrains.com/issue/CPP-17927

        This is a really useful function, but would be significantly more useful if it can be made to run faster on larger sets of tests.

  8. Helge Penne says:

    November 7, 2019

    This EAP crashed abruptly while working (suddenly just closed due to an internal exception):
    https://youtrack.jetbrains.com/issue/CPP-17952

    Show probably be looked into before releasing.

    • Anastasia Kazakova says:

      November 7, 2019

      We’ll definitely take a look. Thanks. And we’ll proceed in the ticket if we need some extra info from you

  9. Roman Popov says:

    November 7, 2019

    Probably you would also want to fix CPP-17819 before Release. Each time I restart Clion all C++ files have “Context: “. To fix it I need to reload CMake project after each Clion restart. I have a stable reproducer for this issue, I can’t share my code, but I can provide any dumps you need.
    But I consider it as minor : I just trained myself to reload CMake project after each restart. Overall benefits of Ninja overweight this issue for me.

    • Anastasia Kazakova says:

      November 8, 2019

      We do remember about this one, but for now it’s not clear what’s going on there for you. Still under investigation. We’ll request additional info when we can figure out what can help us in this investigation. Thanks

  10. Helge Penne says:

    November 8, 2019

    This EAP has some serious freeze issues:
    https://youtrack.jetbrains.com/issue/CPP-17961

    Please fix this before releasing!

  11. Nick Brook says:

    November 30, 2019

    Why was this an image?

    if(“${CMAKE_C_COMPILER_ID}” MATCHES “(Apple)?[Cc]lang” OR “${CMAKE_CXX_COMPILER_ID}” MATCHES “(Apple)?[Cc]lang”)
    message(“Building with llvm Code Coverage Tools”)
    set(CMAKE_CXX_FLAGS “—fprofile—instr—generate —fcoverage—mapping”)
    elseif(CMAKE_COMPILER_IS_GNUCXX)
    message(“Building with lcov Code Coverage Tools”)
    set(CMAKE_CXX_FLAGS “–coverage”)
    endif()

    • Anastasia Kazakova says:

      November 30, 2019

      Several technical reasons, mostly related to the digest blog functionality. The code snippets get broken when rendered in the emails from the blog.

  12. Andrew Mackenzie says:

    January 27, 2020

    Awesome! Great integration!
    I am using with rust on Clion and works great.

    QUESTION
    Is there a way to configure it to not collect/measure coverage on specific modules?
    In particular I would like to focus on the code not the tests and hence exclude coverage from mod test {} module I have in the same source files as they are testing

    • Mikhail Chernyavsky says:

      January 28, 2020

      Hello! IntelliJ Rust uses -Zprofile flag (https://github.com/rust-lang/rust/issues/42524) to generate profile output and grcov (https://github.com/mozilla/grcov) to collect code coverage data from it. These tools are quite experimental and lack some functionality, including the one you are asking for. Currently, the plugin doesn’t provide the ability to exclude specific modules from coverage, but if this feature appears in -Zprofile or grcov, we will be happy to support it.

  13. j says:

    February 13, 2020

    If you remove coverage from the “choose coverage suit to display” dialogue. how do you add it back?
    In other words, where the heck are the coverage files located?
    I see some gcov files under .dir, but selecting them does not activate the OK button in the dialogue to add them.
    Thanks.

    • Vladimir Plyashkun says:

      February 14, 2020

      Hi! Unfortunately this dialog has restriction: you can choose only single file (which is incorrect in case of gcov, since it generates .gcov file for each object file) and there is no possibility to select multiple files here.

      We store coverage output under Generation Path (e.g. cmake-build-debug-cygwin/coverage), but each new coverage execution deletes previous results.

      Moreover, there are some technical issues for a moment with loading coverage result from files (especially on Windows), i’ve created ticket for it: https://youtrack.jetbrains.com/issue/CPP-19077

  14. Steve Magness says:

    April 16, 2020

    Is there somewhere I should be setting a path to .gcda files? I’m getting:

    “Failed to generate coverage data: .gcda files not found
    Note that -fprofile-dir and -fprofile-note compiler options are not yet supported”

    but I can see *.gcda files are being generated. Note I’m using Remote Develop, with CLion installed on a Windows machine and building/running on a RaspberryPi with gcc (Raspbian 8.3.0-6+rpi1) 8.3.0

    • Anastasia Kazakova says:

      April 16, 2020

      Unfortunately, coverage is not yet supported in Remote Mode: https://youtrack.jetbrains.com/issue/CPP-17709
      We’ll probably need to update the message now to be more clear about it. Thanks for pointing.

  15. Kristian Lippert says:

    May 18, 2020

    Q: Saving Code Coverage Output to file?

    Hi,
    We are running CLion (2020.1) and C++ on MacOS.
    We can see code coverage for each unit test and combine them, but how do I save the result as XML and/or HTML/PDF?

    • Vladimir Plyashkun says:

      May 18, 2020

      Hi! Unfortunately there is no option to export coverage result from CLion to HTML/PDF/XML or any other format.

      • Kristian Lippert says:

        May 19, 2020

        Hi!,
        I could actually mark and copy to MS Excel. And that was ok here.
        Best Regards,
        Kristian

  16. Kristian Lippert says:

    May 18, 2020

    Q: See Coverage Build Steps from CI env

    Hi,
    We are running CLion (2020.1) and C++ on MacOS.
    We can see code coverage for each unit test in CLion tool but would like to run this from a CI build server (as we do with the tests anyway).
    I have added the flags to the CMakefile and can see with “make VERBOSE=1” that the flags are there for my tests (but not the gtest-files). But I get no output at all after running unit-test from cmd line.
    Which files should I look for?
    Where will they go?
    Best Regards,
    Kristian

    • Vladimir Plyashkun says:

      May 18, 2020

      Am i right that you are trying to get coverage data from command line outside of CLion?
      Is so, it depends on which compiler and which compiler options do you use.
      For example, if target application compiled by GCC with --coverage flag, it generates “.gcda” files (usually placed in same directory with source binary) after execution. But to get meaningful and text readable data you need to run “gcov” tool on these files manually, see: https://gcc.gnu.org/onlinedocs/gcc/Invoking-Gcov.html#Invoking-Gcov

      For clang – there is similar approach. You need to execute “llvm-profdata” and “llvm-cov” on generated “.profraw” files.
      More detailed information can be found here: https://clang.llvm.org/docs/SourceBasedCodeCoverage.html

Subscribe

Subscribe for updates