Working with Makefiles in CLion using Compilation DB

One of our most frequent feature requests for CLion is the ability to use project or build systems other than CMake. There are many good reasons for needing to use something different – but if you could only pick one we still feel CMake is the best choice. That’s not because we think it is perfect (although it’s worth noting that some of the issues that people have with it may be addressed by focusing on “Modern CMake”). Instead it’s because CMake is in very widespread use – especially in the open-source world, where it has become the de-facto standard.

CMake also has the advantage that it is a meta-build system. It can generate build files for other major build systems. In fact that is really the only way works, although CLion uses it in a way that mostly hides that from you by default (generating Makefiles behind the scenes).

While this is a great starting point, we do want to support other build systems and project formats and work has been progressing over the last couple of releases towards fully decoupling CMake. The initial results of that ongoing work have been: support for Cargo (for Rust projects) in 2018.1, a third-party plug-in for Bazel, and, in 2018.2, Gradle C++ projects and the ability to open Compilation Database JSON files. This is just the start, but is also a lot more useful already than you may realise. To find out why we need to look at what a Compilation Database actually is.

What is a Compilation Database?

In the most general terms a Compilation Database is simply the collection of all information needed to compile a source file, or set of source files – on a file-by-file basis. That includes the filename, all flags and other arguments to be passed to the compiler and the directory to invoke the compiler from (this affects relative paths).

More practically a Compilation Database is most often serialised as a JSON file and comes out of the Clang tooling world. It has become an important mechanism for interoperating between different tools that need to know how code should be compiled – even if not for compilation itself.
Compilation DB JSON files can be generated by an increasing number of build systems and other tools.

What does this mean for CLion users?

Now that CLion can open Compilation DB JSON files it means it can effectively understand projects from many more build systems.
This gives CLion full code-completion, static analysis, navigation and even refactoring on such projects. For 2018.2 we don’t yet have build (except for individual files) or debug capabilities – but these are planned for an upcoming release.

To see just how useful this is, we’ll work through a couple of examples.

Compilation Database Generator

Before we get to the examples themselves, we’ll need to install a tool that can generate Compilation DB JSON files from Makefiles. The tool we’ll use for this tutorial is compiledb. It’s written in Python, so the simplest way to get it is to use pip.

$ pip install compiledb

(you may need to do this with sudo)

Example 1: OpenSSL

OpenSSL is a large, mature, well-known and widely used, open source code base. It’s written mostly in C and has an incredibly complex build system using generated make files.

If you want to follow along you can find the source hosted on GitHub. Taking a fresh clone of this repository you’ll notice it has no Makefiles yet. Like many large, complex, make-based projects you must first run a configuration step that generates the Makefile. Full instructions are in the text file INSTALL in the root of the repository. For Unix-based OSs (including Linux and Mac OS/X/ macOS), the first step is to execute the configure step in a shell:

$ ./config

(Note there are some prerequisites, such as having Perl 5 with core modules – see the INSTALL file for more details).

This runs briefly and, hopefully, completes with no errors, leaving a Makefile in the same directory. If you open the Makefile you’ll see it is not really for human reading – and certainly not for modifying. But now we can use it to give CLion insight into how the codebase should be understood.

To do that we now need to generate a Compilation DB JSON file:

$ compiledb -n make

(passing -n stops make from performing the actual build. If we omit it it will build the whole of OpenSSL at the same time).

This should tell us that it is “Writing compilation database with [some number of] entries to compile_commands.json”.

We can now open the directory in CLion.

Open directory

CLion will detect the compile_commands.json file and look in there for its project information. You may see it think about it for a few seconds, then show the “Compiler Info” tab on the Build tool window, telling you that everything finished ok (there were any errors or warnings you’ll see them in the “Sync” tab).

Compiler Info tab in Build tool window

We can now explore the codebase in CLion. What better way to start than to do a “Search Everywhere” (hit the Shift key twice) and type aria.c to bring up one of the source files. Assuming indexing has finished (which shouldn’t take long) we should instantly find this file.

If we open that and browse through you’ll see full syntax highlighting. Half of this file is #ifdefed out. The second half should lack highlighting and code insight features as OPENSSL_SMALL_FOOTPRINT is not defined (if you used the default configuration).

Find one of the functions (in the enabled first half), e.g. aria_encrypt. If you “Find Usages” on this it will find all the usages in other files in the project.

Find Usages

So now we can finally find our way around large Makefile codebases like OpenSSL.
But because OpenSSL’s make files are generated, we shouldn’t modify them ourselves. Let’s look at a simpler example to see what happens when we do.

Example 2: Modifying Makefiles

For this example we’re going to consider a very simple Makefile. It doesn’t matter what the code does. What’s important is that it has two or more source files, but is simple enough we can easily see what the Makefile is doing. Here’s a starting point:

Notice here we have two source files. To keep things even simpler we’ll leave out discussion of how to automatically extract header file dependencies.

Now, as the project grows you’ll probably want to add additional source files – or make other changes to this make file. If we do that it’s going to get out of sync with the Compilation DB that CLion is working off.

We can, of course, drop to the terminal and invoke compiledb again every time this happens. But that’s tedious and can be error prone. In any case we’re programmers. We like to automate things!

That’s where CLion’s File Watchers come in. File Watchers are a powerful feature in CLion that allows us to trigger external tools or scripts in response to changes in specific files.

Before we get started we’ll need to make sure that CLion is aware of Makefiles. We can do that by adding the Makefile support plug-in (Preferences / Settings ->Plugins->Browse Repositories, “Makefile support”)

Browsing plugin repositories

Once installed (you’ll need to restart CLion) if you open the Makefile you’ll see it is now syntax highlighted. But that’s not why we needed the plugin.

With the Makefile open, go to preferences/settings again, and this time find “File Watchers” (under “Tools” – or just start typing “Watch”). Click the + at the bottom left and select “” from the list of templates.

Give the new File Watcher a name, like “Makefile watcher”, and from the “File type” dropdown select “GNU Makefile” (start typing “make” on the field to find it quickly). That’s what we needed the plugin for.

Fill out the rest of the fields as below (or set “Show console” to your choice):

Creating a new File Watcher

After you click OK, CLion will start monitoring the Makefile and, if it changes – whether from inside the IDE or externally – it will trigger compiledb to run again, regenerating the Compilation DB. To verify this, open the compile_commands.json file side-by-side with the Makefile. Add or remove source files from the srcfiles variable and watch as compile_commands.json is regenerated automatically.

In the end, make sure to set the Use auto-import checkbox in Settings / Preferences | Build, Execution, Deployment | Compilation Database. With this option enabled, the project will be automatically reloaded for every change in compile_command.json.

Download CLion

Something to build on

We’re not stopping here. We’ll continue working towards supporting more build systems and project formats in more integrated ways. But as of now there are some very useful and valuable options you can take advantage of today.

About Phil Nash

Developer Advocate at JetBrains for CLion, AppCode and ReSharper C++
This entry was posted in Tips'n'Tricks and tagged , , , . Bookmark the permalink.

39 Responses to Working with Makefiles in CLion using Compilation DB

  1. Eugene says:

    Will this eventually be available in Android Studio?

  2. Roman says:

    Nice idea with file watchers

  3. Ed Halferty says:

    It’s been awhile since I used MSVC (it’s seems bloated and slow these days – which is why I’m using CLion to begin with), but I think that it’s way of handling projects was spot-on. They just created their own custom project file format instead of using an existing general-purpose build system.
    But I get the sense that most CLion users would prefer to use a non-IDE-specific project file format.
    So I think you’re on the right track: Giving more power to plugins so that when someone eventually makes a great build tool for C++, someone can make a plugin and we can all just use it immediately.

  4. John Murray says:

    This looks really important, but I am having trouble. I am trying to use the Makefile support to execute and debug a Makefile generate target.

    1. I successfully created a Makefile project and built a target based on the Makefile.
    2. I am not able to debug the executable. This is where I am stuck.
    3. I created a compile_commands.json file, which clion recognized.
    4. I do not see a “preferences” setting.
    5. I cannot find a “File Watchers” under preferences.
    6. When trying to create an executable for the Makefile configuration, there is not such field.
    7. In trying to create a configuration from an application template, I always get “error running ‘Execute’ [the name I gave to the application template]: Cannot run ‘Execute’ on ”

    • Anastasia Kazakova says:

      Preferences (macOS) = Settings (Win/Lin). I’ve updated the post.
      To work with File Watchers, you first have to install the plugin: https://www.jetbrains.com/help/clion/using-file-watchers.html#ws_file_watcher_before_you_start

      For compilation database format currently building the whole project/running/debugging is not supported, mainly because compilation database lacks the corresponding information. You can only recompile a single file for now, as this command is provided directly in the compilation database.

      • Shalom Elkin says:

        Hi,
        If I understand correctly Anastasia’s answer, the file watcher and compiledb are almost useless. The real value of the IDE is the debuuger, not the builder, as most people spend most of their time hunting failures and misbehaviours, not in constructing very complex build trees.

        I am looking for a very simple use case : two c files into an libXXX.so, one main file to exercise it, build and debug. Can it be done using compiledb
        with/without file watchers (which are really a gimmick???)

        Thanks,

        Shalom Elkin

        • Anastasia Kazakova says:

          I agree that debugging is a very important point. However, still many users looking for a chance to use CLion as a smart editor, which provide refactoring abilities, navigation, code analysis running on-the-fly and suggesting quick fixes. All these goodies can be provided via compdb.

          Still, of course, we do work on the ability to provide build/run/debug commands for compdb. As for now, you can create a fake CMake project and a CMake target, then in the corresponding CLion’s run/debug configuration you can provide your binary to debug. In next release we plan to make an ability to create run/debug configuration w/o CMake target.

          • wendong song says:

            After generating compile_commands.json file, then open the project with clion. But there is no binary file. Is the binary generated using the ” make” command?

            Add the binary file to run/debug configuration, Can I see the variable information in the code during debugging?

          • Anastasia Kazakova says:

            Compilation database format for now is supported w/o build/run/debug. So you have to build manually and if you need a debugger, then unfortunately it still requires some CMake configuration (https://youtrack.jetbrains.com/issue/CPP-14779).

    • Abel says:

      I come accross a solution using this CMakeLists.txt, then adding the executable manually to configuration targets

      cmake_minimum_required(VERSION 3.5 FATAL_ERROR)
      project( VERSION )

      macro (add_target name)
      set(default_opt BUILD_DIR=${CMAKE_BINARY_DIR}/${name})
      add_custom_target(${name}
      WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}
      COMMAND rm -rf compile_commands.json
      COMMAND compiledb -n make ${default_opt} ${ARGN}
      COMMAND make ${default_opt} ${ARGN})
      endmacro()

      add_target( ..)
      ..

  5. wendong song says:

    I used the ”compiledb -n make“ command to generate the compile_commands.json file, but when I opened the folder with clion, I still prompted error “CMakeLists.txt not found.”

    two files in my project ,helloworld.cpp and makefile.This problem has troubled me for a long time

    • wendong song says:

      ————–makefile————–
      hello: hello.o
      g++ -o hello hello.o

      hello.o: hello.cpp
      g++ -c hello.cpp
      —————hello.cpp————-
      #include
      int main(){
      int a=10;
      for(int i =0;i!=200;++i){
      a=i;
      printf(a);
      }
      printf(“\r\nHello World”);
      }

    • Anastasia Kazakova says:

      Please open compilation database json file (not folder) and ask CLion to open as project when it asks.

  6. wendong song says:

    When you open the project, also need to p erform the following steps
    File–>open–>compile_commands.json ?

  7. laven says:

    I ran compliedb for openssl, using command: compiledb -n make But I got the following error:

    ## Processing build commands from
    make: *** [Makefile] Error 1
    make: Failed to remake makefile ‘Makefile’.
    ## Loaded compilation database with 2941 entries from compile_commands.json
    ## Writing compilation database with 0 entries to compile_commands.json
    ## Done.

    Anyone know how to fix it? I also tried Bear, it can generate the compile DB, but Clion failed to parse it.

    • Semyon Kolton says:

      Hi,
      Can’t help you right now with compiledb, but maybe I can help with Bear generated compilation database. What are the parsing errors reported by CLion? Also, can you post a part of you compile_commands.json?
      Maybe we can move this discussion to https://youtrack.jetbrains.com/issues/CPP

      • laven says:

        Thanks for your quick reply! I checked again. I guess the reason is that I used a clang wrapper as CC when configuring OpenSSL. When I used default clang, it works well. This is a problem of Bear installed through pip. It seems the newest version of it on Github supports a new –use-cc option to specify the used cc.

  8. Ravindra Rao says:

    Autocomplete suggestions from standard header files does not seem to work in this case, or am I missing somethiing?

    • Anastasia Kazakova says:

      Maybe you can report a particular case/sample to the tracker? We’ll check. My assumption is that it should work.

  9. Rhys says:

    This works for me for some Makefile based projects, however I’m facing some issues with others. If the Makefile that compiles the source is not at the root of the project, then this didn’t work out well for me. Also if there are sub-libraries built by different Makefiles, then I didn’t see a way of adding those into the compilation DB.

  10. Amith RC says:

    Hello,

    Need help.

    I am trying to build a PTPD source from my professor gitlab.

    Here is the Link.

    https://gitlab.cs.unh.edu/rbartos/ptpd

    I followed the article. Here is what I did.

    git clone https://gitlab.cs.unh.edu/rbartos/ptpd.git
    autoreconf -vi
    ./configure
    compiledb -n make

    imported the project (This time no error and index completed) I was happy thus far.

    After that I do not see anything, no build Tab ( No compiler info tab) When I clicked on Build option, it says Nothing here.
    I clicked on a find usage to see the function usage, it says no usage found in project or libraries.

    on Top it says CMake project is not loaded.

    My question is how to fix this? How can build the project using CLion?

    Please help.

  11. Amith RC says:

    When I open the JSON file via open, it does not ask me whether I need to open a project. It just opens the JSON file and the project name is the same as the JSON file.

  12. Martin says:

    Would be nice if compiledb actually produced any output. My json file is empty. I pretty much just used the simple example above but had to add a line to tell make how to build C files or it would fail. I’m guessing it was trying to use gcc instead of vbcc?

    appname := screentest

    CC := vc
    CFLAGS := -c99 +aos68k -I$(NDK_INC)
    LDFLAGS := -lamiga -lauto

    srcfiles := screentest.c
    objects := $(patsubst %.c,%.o, $(srcfiles))

    OUTPATH := ../hd0

    all: $(appname)

    %.o : %.c
    $(CC) $(CFLAGS) -c $< -o $@

    $(appname): $(objects)
    $(CC) $(CFLAGS) $(LDFLAGS) -o $(OUTPATH)/$(appname) $(objects)

    clean:
    rm -f $(objects)
    rm -f $(OUTPATH)/$(appname)

    • Anastasia Kazakova says:

      How are you actually generating the compilation database? Do you call compiledb-generator (https://github.com/nickdiego/compiledb-generator) or smth?

      • Martin says:

        As per the above blog “compiledb -n make”. However, after digging into it, none of the tools that I could find recognise anything other than gcc or clang so they don’t pick up on ‘vc’ as the compiler which is a shame.

        Given I currently only have a single file I created the son file manually but then given Clion also only supports gcc or clang, IT then complains. I can trick it by telling it I’m really using gcc (when I’m not) but then I get other errors.

        Given I can happily build and clean my project from the makefile it would have been nice to get CLion to handle the IntelliSense by picking up the include folders which may be old, but they’re just standard C. I managed to get to that point but sadly I use a couple of VBCC specific keywords such as “__chip” to tell it to allocate a structure in chip memory (a particular memory area on the Amiga) and IntelliSense doesn’t know what to do with that and complains that I have errors in my code.

        Very close, but not close enough.

  13. Martin says:

    ??

    VBCC has nothing to do with Microsoft:
    http://www.compilers.de/vbcc.html

    and more specifically to what I’m using:
    http://sun.hasenbraten.de/vbcc/

    • Anastasia Kazakova says:

      Oh, sorry, I misunderstand you. Yes, such compilers are not supported, but if you wrap it in some script, so that it produced the output to this command similar to gcc/clang: clang -x c /dev/null -dM -E, then it might work.

      • Martin says:

        I can trick it into believing it’s using gcc so IntelliSense works (though I think it’s not great I need to do that when I’m not actually asking it to run the compiler). Is there a way to instruct IntelliSense to completely ignore certain words?

        For example, the following line:
        static UWORD __chip copList[] = { blah }

        CLion thinks there is an error because it does not understand “__chip”. If I could tell it to simply ignore that keyword then it would be fine as everything builds OK via make.

  14. Ranz says:

    How could I compile my Makefile-based project with compiledb when I am in a full time remote development? It doesn’t work for me. So, this is still not supported yet?

    • Anastasia Kazakova says:

      Remote toolchain in CLion doesn’t work with compilation database. We plan to have it in the future. Currently it’s with CMake only.

Leave a Reply to Eugene Cancel reply

Your email address will not be published. Required fields are marked *