CLion 2019.3 EAP: Code Coverage and CMake Defaults

Hi,

A new CLion 2019.3 EAP (build 193.4932.12) is now available! Get it from our website, via the Toolbox App, or as a snap package (if you are using Ubuntu). A patch-update for anyone using the previous EAP build will be available shortly.

DOWNLOAD CLION 2019.3 EAP

Key highlights:

Code Coverage

Code coverage in CLion relies on llvm-cov/gcov integration to perform and show you statements coverage measuring for your code. In general, statements coverage answers the question, "Was this statement executed during the configuration run?". The configuration can be either a unit tests run or a regular configuration run.

How to launch a configuration with coverage?

To get coverage measurement, you have to pass special coverage compile options. There are a few options depending on what compiler you use and what kind of results you expect:

  • -fprofile-arcs -ftest-coverage or an equivalent --coverage – for GCC (uses gcov) and Clang (uses llvm-cov gcov, gcov-style coverage, which means it will compatible with the gcov tool from version 4.2 of GCC)
  • -fprofile-instr-generate -fcoverage-mapping – for Clang (uses llvm-profdata merge / llvm-cov export, a clang’s instrumentation based profiling).

For example, if you use both GCC and Clang in your project, you may have something like this in your CMakeLists.txt:
coverage_cmake

Now you can run it using the run icon in the left gutter:
coverage_run

Or you can click the “Run … with Coverage” button coverage_icon in the Navigation Bar.

How to read the results

Coverage measurements are located in the Coverage tool window, which shows results per folder (% of files and % of lines covered) and per file (% of lines covered):
coverage_results

When rerunning the coverage analysis, CLion will ask you how you’d like the results presented:

  • Add to active suites – to enrich the current results.
  • Replace active suites – to start from scratch.
  • Do not apply collected coverage data – to simply ignore the current results.

Another way of exploring the results is to check the color indication in the left gutter in the editor.
coverage_indication_gutter

Click the gutter area to see how many times the line was hit.

Configuration and known issue

Coverage settings, including custom paths to gcov / llvm-cov / llvm-profdata tools and more, can be found in Settings/Preferences | Build, Execution, Deployment | Coverage:
coverage_settings

A few important known issues are worth mentioning:

  • Code coverage doesn’t work with Ninja generator (CPP-17864). We’ll do our best to address this issue before the release.
  • Code Coverage is not supported for the remote toolchain (CPP-17709).
  • Generate Coverage Report – an action you may be familiar with if you use IntelliJ IDEA – is not available yet (CPP-17710).

Other bugs and requests are linked to the parent ticket, CPP-9678. Please try the EAP build with code coverage support and share your feedback with us!

CMake defaults

If you have to work with a lot of new projects in CLion, you may want to simplify the process of configuring CMake Profiles, for example by configuring a common pattern for the CMake generation folder.

Starting with this EAP, you can configure one or several default CMake Profiles which will be used for all your new CLion projects! Use File | Other Settings | Settings/Preferences for New Projects…:
cmake_settings

You can set the default:

  • Generator used in CMake
  • Options passed to CMake
  • Toolchain
  • Build type
  • Environment
  • Generation path
  • Build options

When a new project is started, these default CMake Profiles will be used. Note that currently, this doesn’t apply when File | New CMake Project from Sources is used (CPP-17686).

That’s it for today! The full release notes are available here.

DOWNLOAD CLION 2019.3 EAP

Your CLion Team
JetBrains
The Drive to Develop

This entry was posted in Announcement, Early Access Preview and tagged , , , , . Bookmark the permalink.

31 Responses to CLion 2019.3 EAP: Code Coverage and CMake Defaults

  1. Martin Pecka says:

    Nice, the default CMake profiles might come handy.

    When I set an absolute generation path, can I use some variables in the input? If I for example wanted to put all CLion build directories under /data/clion/{project-name}/cmake-build-debug …

  2. mbalabin says:

    Reading a local .lldbinit feature is a nice little feature! But it seems that it has not been released yet. The corresponding ticket (https://youtrack.jetbrains.com/issue/CPP-17608) is still in progress.

    • Eldar Abusalimov says:

      I’m afraid I owe an apology on this one. The feature was not included into the released build indeed. I got switched to some other tasks, and forgot to put the necessary changes into the release branch. Sorry for the confusion.

  3. Henry says:

    oh my god! Coverage!!! This is awesome!

  4. Jonathan Ross says:

    Like Henry, I’m happy to see code coverage added to CLion! (I have unfortunately not yet gotten it to work: my coverage results are empty.)

    It seems error-prone to have to add coverage compilation flags to my CMake projects by hand to get it to work. Are there any plans to automate this/add the flags behind the scenes?

    • Anastasia Kazakova says:

      Hi,

      Feel free to issue a bug if case coverage still fails, We’ll request some additional logs and info and will try to help.

      As for the automation, we are still unsure here. In general, we prefer not to interfere in the CMake process from the IDE’s side, which means not adding any flags by the IDE itself. But we understand the current workflow is not very convenient and user-friendly. So we are thinking about the solution.

      • Jonathan Ross says:

        I managed to get the code coverage working: it turned out that gcov was aliased to llvm-cov on my machine (Mac). When I configured the path explicitly it worked great. My one remaining gripe would be that the coverage analysis failed silently when I had it configured incorrectly.

        Regarding automation / not interfering with the CMake process: one solution that springs to mind, would be for you to intercept at the toolchain level: wrap the compiler with a script that appends the relevant flags.

        • Anastasia Kazakova says:

          Good to know it works! We’ll see if we can provide better logging.

          Regarding the wrapping, it’s indeed one of the solutions. But also has its drawbacks.

  5. David Zemon says:

    Finally got around to trying this today. Very excited to see the familiar coverage GUI back in my toolbox, after moving from Java to C/C++ a couple years ago.

    One big disappointment though: it doesn’t seem possible to tell CLion that multiple coverage reports need to be combined together. This makes it far more difficult to get the whole-project-picture of code coverage.

    • Anastasia Kazakova says:

      What do you mean by combining the reports? When you rerun the coverage, CLion asks if you’s like to add to the current report, drop results or start a new one. Do you mean smth different here?

  6. Andrew Smith says:

    It looks like the new “Remote GDB Server” still requires that one have a project that builds locally, and runs the GDB client locally. My project cannot build with CLion locally or remotely; I build it outside of CLion. Is there a way to remotely debug over SSH without a local build (similar to how it works with a remote toolchain, but just running a remote process built through an external build system)?

    • Anastasia Kazakova says:

      For such cases we have GDB Remote Debug configuration: https://www.jetbrains.com/help/clion/2019.3/remote-debug.html
      You’ll have to manually start the gdbserver, provide CLion with the path mappings and symbols for the debug (this is necessary as you build outside of CLion).

      • Andrew Smith says:

        The problem with that is that this would require transferring symbols every time I debug. My symbols are over 1gb, which is prohibitive to transfer every time. Your SSH debugging with a remote tool chain (using gdb over SSH) works perfectly, except that CLion only supports that with projects that build remotely in CLion.

        It seems like you have all of the hard stuff done for GDB over SSH, and the only thing missing is a simple UI entry point to support it for projects that build outside of CLion. Or is there another way to do this that I’m missing?

        • Anastasia Kazakova says:

          Do you still build locally? Or is it a 3-rd machine?

          • Andrew Smith says:

            I build on a remote machine (the same machine I want to debug on). The code won’t build locally, both because of a different build system and different local OS. I have confirmed that SSH debugging works with a CLion remote tool chain project (where CLion builds the project remotely); there just doesn’t seem to be a way to do it in the UI outside of a remote tool chain project (which requires CLion to do the build).

          • Anastasia Kazakova says:

            Ok, but not sure I get the problem why you can’t use the Full Remote mode, which builds on a remote machine. Indeed it’s done from the CLion. Why it doesn’t work for you? Do you have some custom build different from CMake?

          • Andrew Smith says:

            Yes, we use a completely different build system (with distributed caching of intermediate artifacts/etc).

          • Anastasia Kazakova says:

            Ok, but CLion has to get the build symbols for the debug somehow. Either it gets them when building on its own, or you upload it to the host where CLion is on your own. You can probably configure some rsync or similar to do that for you automatically.

  7. Helge Penne says:

    Integrated coverage is really nice. I’ve given it a test run and here is what I’ve found so far:
    – It works really well for small tests executables.
    – When I ran it on a test executable with a few thousand tests in it, CLion was stuck in processing for at least 15 minutes and eventually failed on an out-of-memory error.
    – Coverage processing seems single treaded, which is a pity when it takes quite some time to process large sets of tests. It seems you should work more on speed and memory usage. The current implementation will not be good enough for large projects.
    – I do not get the question about add/drop/new etc. that is mentioned on the blog when I re-run tests in coverage mode. This seems like a bug.

    The slow processing and out-of-memory failure is sad. This was with clang. Is there any hope of better results with a gcc toolchain? Can you improve on this?

  8. Helge Penne says:

    This EAP crashed abruptly while working (suddenly just closed due to an internal exception):
    https://youtrack.jetbrains.com/issue/CPP-17952

    Show probably be looked into before releasing.

    • Anastasia Kazakova says:

      We’ll definitely take a look. Thanks. And we’ll proceed in the ticket if we need some extra info from you

  9. Roman Popov says:

    Probably you would also want to fix CPP-17819 before Release. Each time I restart Clion all C++ files have “Context: “. To fix it I need to reload CMake project after each Clion restart. I have a stable reproducer for this issue, I can’t share my code, but I can provide any dumps you need.
    But I consider it as minor : I just trained myself to reload CMake project after each restart. Overall benefits of Ninja overweight this issue for me.

    • Anastasia Kazakova says:

      We do remember about this one, but for now it’s not clear what’s going on there for you. Still under investigation. We’ll request additional info when we can figure out what can help us in this investigation. Thanks

  10. Helge Penne says:

    This EAP has some serious freeze issues:
    https://youtrack.jetbrains.com/issue/CPP-17961

    Please fix this before releasing!

Leave a Reply

Your email address will not be published. Required fields are marked *