Working with Makefiles in CLion using Compilation DB

One of our most frequent feature requests for CLion is the ability to use project or build systems other than CMake. There are many good reasons for needing to use something different – but if you could only pick one we still feel CMake is the best choice. That’s not because we think it is perfect (although it’s worth noting that some of the issues that people have with it may be addressed by focusing on “Modern CMake”). Instead it’s because CMake is in very widespread use – especially in the open-source world, where it has become the de-facto standard.

CMake also has the advantage that it is a meta-build system. It can generate build files for other major build systems. In fact that is really the only way works, although CLion uses it in a way that mostly hides that from you by default (generating Makefiles behind the scenes).

While this is a great starting point, we do want to support other build systems and project formats and work has been progressing over the last couple of releases towards fully decoupling CMake. The initial results of that ongoing work have been: support for Cargo (for Rust projects) in 2018.1, a third-party plug-in for Bazel, and, in 2018.2, Gradle C++ projects and the ability to open Compilation Database JSON files. This is just the start, but is also a lot more useful already than you may realise. To find out why we need to look at what a Compilation Database actually is.

What is a Compilation Database?

In the most general terms a Compilation Database is simply the collection of all information needed to compile a source file, or set of source files – on a file-by-file basis. That includes the filename, all flags and other arguments to be passed to the compiler and the directory to invoke the compiler from (this affects relative paths).

More practically a Compilation Database is most often serialised as a JSON file and comes out of the Clang tooling world. It has become an important mechanism for interoperating between different tools that need to know how code should be compiled – even if not for compilation itself.
Compilation DB JSON files can be generated by an increasing number of build systems and other tools.

What does this mean for CLion users?

Now that CLion can open Compilation DB JSON files it means it can effectively understand projects from many more build systems.
This gives CLion full code-completion, static analysis, navigation and even refactoring on such projects. For 2018.2 we don’t yet have build (except for individual files) or debug capabilities – but these are planned for an upcoming release.

To see just how useful this is, we’ll work through a couple of examples.

Compilation Database Generator

Before we get to the examples themselves, we’ll need to install a tool that can generate Compilation DB JSON files from Makefiles. The tool we’ll use for this tutorial is compiledb. It’s written in Python, so the simplest way to get it is to use pip.

$ pip install compiledb

(you may need to do this with sudo)

Example 1: OpenSSL

OpenSSL is a large, mature, well-known and widely used, open source code base. It’s written mostly in C and has an incredibly complex build system using generated make files.

If you want to follow along you can find the source hosted on GitHub. Taking a fresh clone of this repository you’ll notice it has no Makefiles yet. Like many large, complex, make-based projects you must first run a configuration step that generates the Makefile. Full instructions are in the text file INSTALL in the root of the repository. For Unix-based OSs (including Linux and Mac OS/X/ macOS), the first step is to execute the configure step in a shell:

$ ./config

(Note there are some prerequisites, such as having Perl 5 with core modules – see the INSTALL file for more details).

This runs briefly and, hopefully, completes with no errors, leaving a Makefile in the same directory. If you open the Makefile you’ll see it is not really for human reading – and certainly not for modifying. But now we can use it to give CLion insight into how the codebase should be understood.

To do that we now need to generate a Compilation DB JSON file:

$ compiledb -n make

(passing -n stops make from performing the actual build. If we omit it it will build the whole of OpenSSL at the same time).

This should tell us that it is “Writing compilation database with [some number of] entries to compile_commands.json”.

We can now open the directory in CLion.

Open directory

CLion will detect the compile_commands.json file and look in there for its project information. You may see it think about it for a few seconds, then show the “Compiler Info” tab on the Build tool window, telling you that everything finished ok (there were any errors or warnings you’ll see them in the “Sync” tab).

Compiler Info tab in Build tool window

We can now explore the codebase in CLion. What better way to start than to do a “Search Everywhere” (hit the Shift key twice) and type aria.c to bring up one of the source files. Assuming indexing has finished (which shouldn’t take long) we should instantly find this file.

If we open that and browse through you’ll see full syntax highlighting. Half of this file is #ifdefed out. The second half should lack highlighting and code insight features as OPENSSL_SMALL_FOOTPRINT is not defined (if you used the default configuration).

Find one of the functions (in the enabled first half), e.g. aria_encrypt. If you “Find Usages” on this it will find all the usages in other files in the project.

Find Usages

So now we can finally find our way around large Makefile codebases like OpenSSL.
But because OpenSSL’s make files are generated, we shouldn’t modify them ourselves. Let’s look at a simpler example to see what happens when we do.

Example 2: Modifying Makefiles

For this example we’re going to consider a very simple Makefile. It doesn’t matter what the code does. What’s important is that it has two or more source files, but is simple enough we can easily see what the Makefile is doing. Here’s a starting point:

Notice here we have two source files. To keep things even simpler we’ll leave out discussion of how to automatically extract header file dependencies.

Now, as the project grows you’ll probably want to add additional source files – or make other changes to this make file. If we do that it’s going to get out of sync with the Compilation DB that CLion is working off.

We can, of course, drop to the terminal and invoke compiledb again every time this happens. But that’s tedious and can be error prone. In any case we’re programmers. We like to automate things!

That’s where CLion’s File Watchers come in. File Watchers are a powerful feature in CLion that allows us to trigger external tools or scripts in response to changes in specific files.

Before we get started we’ll need to make sure that CLion is aware of Makefiles. We can do that by adding the Makefile support plug-in (Preferences / Settings ->Plugins->Browse Repositories, “Makefile support”)

Browsing plugin repositories

Once installed (you’ll need to restart CLion) if you open the Makefile you’ll see it is now syntax highlighted. But that’s not why we needed the plugin.

With the Makefile open, go to preferences/settings again, and this time find “File Watchers” (under “Tools” – or just start typing “Watch”). Click the + at the bottom left and select “” from the list of templates.

Give the new File Watcher a name, like “Makefile watcher”, and from the “File type” dropdown select “GNU Makefile” (start typing “make” on the field to find it quickly). That’s what we needed the plugin for.

Fill out the rest of the fields as below (or set “Show console” to your choice):

Creating a new File Watcher

After you click OK, CLion will start monitoring the Makefile and, if it changes – whether from inside the IDE or externally – it will trigger compiledb to run again, regenerating the Compilation DB. To verify this, open the compile_commands.json file side-by-side with the Makefile. Add or remove source files from the srcfiles variable and watch as compile_commands.json is regenerated automatically.

In the end, make sure to set the Use auto-import checkbox in Settings / Preferences | Build, Execution, Deployment | Compilation Database. With this option enabled, the project will be automatically reloaded for every change in compile_command.json.

Download CLion

Something to build on

We’re not stopping here. We’ll continue working towards supporting more build systems and project formats in more integrated ways. But as of now there are some very useful and valuable options you can take advantage of today.

About Phil Nash

Developer Advocate at JetBrains for CLion, AppCode and ReSharper C++
This entry was posted in Tips'n'Tricks and tagged , , , . Bookmark the permalink.

22 Responses to Working with Makefiles in CLion using Compilation DB

  1. Eugene says:

    Will this eventually be available in Android Studio?

  2. Roman says:

    Nice idea with file watchers

  3. Ed Halferty says:

    It’s been awhile since I used MSVC (it’s seems bloated and slow these days – which is why I’m using CLion to begin with), but I think that it’s way of handling projects was spot-on. They just created their own custom project file format instead of using an existing general-purpose build system.
    But I get the sense that most CLion users would prefer to use a non-IDE-specific project file format.
    So I think you’re on the right track: Giving more power to plugins so that when someone eventually makes a great build tool for C++, someone can make a plugin and we can all just use it immediately.

  4. John Murray says:

    This looks really important, but I am having trouble. I am trying to use the Makefile support to execute and debug a Makefile generate target.

    1. I successfully created a Makefile project and built a target based on the Makefile.
    2. I am not able to debug the executable. This is where I am stuck.
    3. I created a compile_commands.json file, which clion recognized.
    4. I do not see a “preferences” setting.
    5. I cannot find a “File Watchers” under preferences.
    6. When trying to create an executable for the Makefile configuration, there is not such field.
    7. In trying to create a configuration from an application template, I always get “error running ‘Execute’ [the name I gave to the application template]: Cannot run ‘Execute’ on ”

    • Anastasia Kazakova says:

      Preferences (macOS) = Settings (Win/Lin). I’ve updated the post.
      To work with File Watchers, you first have to install the plugin: https://www.jetbrains.com/help/clion/using-file-watchers.html#ws_file_watcher_before_you_start

      For compilation database format currently building the whole project/running/debugging is not supported, mainly because compilation database lacks the corresponding information. You can only recompile a single file for now, as this command is provided directly in the compilation database.

      • Shalom Elkin says:

        Hi,
        If I understand correctly Anastasia’s answer, the file watcher and compiledb are almost useless. The real value of the IDE is the debuuger, not the builder, as most people spend most of their time hunting failures and misbehaviours, not in constructing very complex build trees.

        I am looking for a very simple use case : two c files into an libXXX.so, one main file to exercise it, build and debug. Can it be done using compiledb
        with/without file watchers (which are really a gimmick???)

        Thanks,

        Shalom Elkin

        • Anastasia Kazakova says:

          I agree that debugging is a very important point. However, still many users looking for a chance to use CLion as a smart editor, which provide refactoring abilities, navigation, code analysis running on-the-fly and suggesting quick fixes. All these goodies can be provided via compdb.

          Still, of course, we do work on the ability to provide build/run/debug commands for compdb. As for now, you can create a fake CMake project and a CMake target, then in the corresponding CLion’s run/debug configuration you can provide your binary to debug. In next release we plan to make an ability to create run/debug configuration w/o CMake target.

          • wendong song says:

            After generating compile_commands.json file, then open the project with clion. But there is no binary file. Is the binary generated using the ” make” command?

            Add the binary file to run/debug configuration, Can I see the variable information in the code during debugging?

          • Anastasia Kazakova says:

            Compilation database format for now is supported w/o build/run/debug. So you have to build manually and if you need a debugger, then unfortunately it still requires some CMake configuration (https://youtrack.jetbrains.com/issue/CPP-14779).

    • Abel says:

      I come accross a solution using this CMakeLists.txt, then adding the executable manually to configuration targets

      cmake_minimum_required(VERSION 3.5 FATAL_ERROR)
      project( VERSION )

      macro (add_target name)
      set(default_opt BUILD_DIR=${CMAKE_BINARY_DIR}/${name})
      add_custom_target(${name}
      WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}
      COMMAND rm -rf compile_commands.json
      COMMAND compiledb -n make ${default_opt} ${ARGN}
      COMMAND make ${default_opt} ${ARGN})
      endmacro()

      add_target( ..)
      ..

  5. wendong song says:

    I used the ”compiledb -n make“ command to generate the compile_commands.json file, but when I opened the folder with clion, I still prompted error “CMakeLists.txt not found.”

    two files in my project ,helloworld.cpp and makefile.This problem has troubled me for a long time

    • wendong song says:

      ————–makefile————–
      hello: hello.o
      g++ -o hello hello.o

      hello.o: hello.cpp
      g++ -c hello.cpp
      —————hello.cpp————-
      #include
      int main(){
      int a=10;
      for(int i =0;i!=200;++i){
      a=i;
      printf(a);
      }
      printf(“\r\nHello World”);
      }

    • Anastasia Kazakova says:

      Please open compilation database json file (not folder) and ask CLion to open as project when it asks.

  6. wendong song says:

    When you open the project, also need to p erform the following steps
    File–>open–>compile_commands.json ?

  7. laven says:

    I ran compliedb for openssl, using command: compiledb -n make But I got the following error:

    ## Processing build commands from
    make: *** [Makefile] Error 1
    make: Failed to remake makefile ‘Makefile’.
    ## Loaded compilation database with 2941 entries from compile_commands.json
    ## Writing compilation database with 0 entries to compile_commands.json
    ## Done.

    Anyone know how to fix it? I also tried Bear, it can generate the compile DB, but Clion failed to parse it.

  8. Ravindra Rao says:

    Autocomplete suggestions from standard header files does not seem to work in this case, or am I missing somethiing?

    • Anastasia Kazakova says:

      Maybe you can report a particular case/sample to the tracker? We’ll check. My assumption is that it should work.

Leave a Reply

Your email address will not be published. Required fields are marked *