Working with Makefiles in CLion using Compilation DB
One of our most frequent feature requests for CLion is the ability to use project or build systems other than CMake. There are many good reasons for needing to use something different – but if you could only pick one we still feel CMake is the best choice. That’s not because we think it is perfect (although it’s worth noting that some of the issues that people have with it may be addressed by focusing on “Modern CMake”). Instead it’s because CMake is in very widespread use – especially in the open-source world, where it has become the de-facto standard.
CMake also has the advantage that it is a meta-build system. It can generate build files for other major build systems. In fact that is really the only way works, although CLion uses it in a way that mostly hides that from you by default (generating Makefiles behind the scenes).
While this is a great starting point, we do want to support other build systems and project formats and work has been progressing over the last couple of releases towards fully decoupling CMake. The initial results of that ongoing work have been: support for Cargo (for Rust projects) in 2018.1, a third-party plug-in for Bazel, and, in 2018.2, Gradle C++ projects and the ability to open Compilation Database JSON files. This is just the start, but is also a lot more useful already than you may realise. To find out why we need to look at what a Compilation Database actually is.
What is a Compilation Database?
In the most general terms a Compilation Database is simply the collection of all information needed to compile a source file, or set of source files – on a file-by-file basis. That includes the filename, all flags and other arguments to be passed to the compiler and the directory to invoke the compiler from (this affects relative paths).
More practically a Compilation Database is most often serialised as a JSON file and comes out of the Clang tooling world. It has become an important mechanism for interoperating between different tools that need to know how code should be compiled – even if not for compilation itself.
Compilation DB JSON files can be generated by an increasing number of build systems and other tools.
What does this mean for CLion users?
Now that CLion can open Compilation DB JSON files it means it can effectively understand projects from many more build systems.
This gives CLion full code-completion, static analysis, navigation and even refactoring on such projects. For 2018.2 we don’t yet have build (except for individual files) or debug capabilities – but these are planned for an upcoming release.
To see just how useful this is, we’ll work through a couple of examples.
Compilation Database Generator
Before we get to the examples themselves, we’ll need to install a tool that can generate Compilation DB JSON files from Makefiles. The tool we’ll use for this tutorial is compiledb. It’s written in Python, so the simplest way to get it is to use pip.
$ pip install compiledb
(you may need to do this with `sudo`)
Example 1: OpenSSL
OpenSSL is a large, mature, well-known and widely used, open source code base. It’s written mostly in C and has an incredibly complex build system using generated make files.
If you want to follow along you can find the source hosted on GitHub. Taking a fresh clone of this repository you’ll notice it has no Makefiles yet. Like many large, complex, make-based projects you must first run a configuration step that generates the Makefile. Full instructions are in the text file `INSTALL` in the root of the repository. For Unix-based OSs (including Linux and Mac OS/X/ macOS), the first step is to execute the configure step in a shell:
$ ./config
(Note there are some prerequisites, such as having Perl 5 with core modules – see the `INSTALL` file for more details).
This runs briefly and, hopefully, completes with no errors, leaving a Makefile in the same directory. If you open the Makefile you’ll see it is not really for human reading – and certainly not for modifying. But now we can use it to give CLion insight into how the codebase should be understood.
To do that we now need to generate a Compilation DB JSON file:
$ compiledb -n make
(passing `-n` stops make from performing the actual build. If we omit it it will build the whole of OpenSSL at the same time).
This should tell us that it is “Writing compilation database with [some number of] entries to compile_commands.json”.
We can now open the directory in CLion.
CLion will detect the `compile_commands.json` file and look in there for its project information. You may see it think about it for a few seconds, then show the “Compiler Info” tab on the Build tool window, telling you that everything finished ok (there were any errors or warnings you’ll see them in the “Sync” tab).
We can now explore the codebase in CLion. What better way to start than to do a “Search Everywhere” (hit the Shift key twice) and type `aria.c` to bring up one of the source files. Assuming indexing has finished (which shouldn’t take long) we should instantly find this file.
If we open that and browse through you’ll see full syntax highlighting. Half of this file is `#ifdef`ed out. The second half should lack highlighting and code insight features as `OPENSSL_SMALL_FOOTPRINT` is not defined (if you used the default configuration).
Find one of the functions (in the enabled first half), e.g. `aria_encrypt`. If you “Find Usages” on this it will find all the usages in other files in the project.
So now we can finally find our way around large Makefile codebases like OpenSSL.
But because OpenSSL’s make files are generated, we shouldn’t modify them ourselves. Let’s look at a simpler example to see what happens when we do.
Example 2: Modifying Makefiles
For this example we’re going to consider a very simple Makefile. It doesn’t matter what the code does. What’s important is that it has two or more source files, but is simple enough we can easily see what the Makefile is doing. Here’s a starting point:
appname := question CXX := clang++ CXXFLAGS := -std=c++11 srcfiles := main.cpp foo.cpp objects := $(patsubst %.cpp, %.o, $(srcfiles)) all: $(appname) $(appname): $(objects) $(CXX) $(CXXFLAGS) $(LDFLAGS) -o $(appname) $(objects) $(LDLIBS) clean: rm -f $(objects)
Notice here we have two source files. To keep things even simpler we’ll leave out discussion of how to automatically extract header file dependencies.
Now, as the project grows you’ll probably want to add additional source files – or make other changes to this make file. If we do that it’s going to get out of sync with the Compilation DB that CLion is working off.
We can, of course, drop to the terminal and invoke `compiledb` again every time this happens. But that’s tedious and can be error prone. In any case we’re programmers. We like to automate things!
That’s where CLion’s File Watchers come in. File Watchers are a powerful feature in CLion that allows us to trigger external tools or scripts in response to changes in specific files.
Before we get started we’ll need to make sure that CLion is aware of Makefiles. We can do that by adding the Makefile support plug-in (Preferences / Settings ->Plugins->Browse Repositories, “Makefile support”)
Once installed (you’ll need to restart CLion) if you open the Makefile you’ll see it is now syntax highlighted. But that’s not why we needed the plugin.
With the Makefile open, go to preferences/settings again, and this time find “File Watchers” (under “Tools” – or just start typing “Watch”). Click the `+` at the bottom left and select “
Give the new File Watcher a name, like “Makefile watcher”, and from the “File type” dropdown select “GNU Makefile” (start typing “make” on the field to find it quickly). That’s what we needed the plugin for.
Fill out the rest of the fields as below (or set “Show console” to your choice):
After you click `OK`, CLion will start monitoring the Makefile and, if it changes – whether from inside the IDE or externally – it will trigger `compiledb` to run again, regenerating the Compilation DB. To verify this, open the `compile_commands.json` file side-by-side with the Makefile. Add or remove source files from the `srcfiles` variable and watch as `compile_commands.json` is regenerated automatically.
In the end, make sure to set the Use auto-import checkbox in Settings / Preferences | Build, Execution, Deployment | Compilation Database. With this option enabled, the project will be automatically reloaded for every change in compile_command.json.
Something to build on
We’re not stopping here. We’ll continue working towards supporting more build systems and project formats in more integrated ways. But as of now there are some very useful and valuable options you can take advantage of today.
Subscribe to CLion Blog updates
