Dealing with Makefile Projects in CLion: Status Update

Anastasia Kazakova

Update: Makefile projects support is now public in CLion 2020.2.

What request in our tracker has more than 1000 votes, 370 comments, and 800 watchers? You guessed it: Support Makefile projects. This has been a story of interesting findings, semi-automatic workarounds, and a long battle that we still plan to win! If you’d like to get the latest news on this topic, please read on.

• From past to present: the evolution of project models in CLion
• Support for Makefile projects: How-to
  • Option 1: Compiler wrappers
  • Option 2: LD_PRELOAD
  • Option 3: Parse the output of Make
• Support for Makefile projects: CLion’s prototype
  • What could possibly go wrong?
  • Projects we used for testing
• Call for help!

From past to present: the evolution of project models in CLion

As you know, all work in CLion is done within the context of a project. Projects serve as the basis for coding assistance, bulk refactoring, coding style consistency, and other smart features. To understand the project model, CLion collects not only the list of project files, but also compilation flags, header search paths, and project-model specific variables. The IDE can create several resolve configurations aggregating these parameters, which you can switch in the editor when necessary.

5 years ago CLion was launched with CMake-only projects, and we shared the reasoning behind that decision in our blog. Since then, CMake has been growing steadily in the C++ community and finally overtook Visual Studio to become the most popular project model / build system for C++ development.

However, in the world of C++ there is no single standard project model. There are Makefile projects, qmake, msbuild, Bazel, Scons, and many others. Makefile projects are in the top 3, popular in OSS and embedded projects, and widely used for cross-platform C++ development, which makes them the best candidate to add to CLion.

On our journey to supporting Makefile projects, we’ve taken the following steps:

• Open a folder in CLion. With no project model in place, CLion treats all the code on a very basic level and can’t provide any smart coding assistance (like Go to Symbol or refactorings).
• Compilation database support. This is a truly universal project format, which can be generated from any project model, including any custom one. CLion provides full coding assistance for such projects, and often that’s the easiest way to open an arbitrary project in CLion.
• Support for custom build targets and custom Run/Debug configurations. We added this to enrich the compilation database or folder-based projects experience, as build/clean commands can then be used by CLion and such applications can be run and debugged.

While it’s now possible to build, run and debug any projects (including Makefile-based) in CLion by using Custom Build Targets, it requires an accurate manual setup. The CMake project experience is much better here – CLion automatically detects available targets to build, and executables to run/debug. We’re inclined to provide this experience for Makefile-based projects as well, however, this requires a lot of heuristics and tuning. In the meantime, you can use existing Makefile plugin to run make targets (note that it doesn’t allow running or debugging executables, you still should use Custom Build Targets for this).

This brought us to the workflow where users work around Makefile projects with the help of File Watchers and the compilation database. The main pain-point is that you have to install the tools to extract the compilation database from your Makefiles. However, the workflow is universal and works smoothly for any project model not supported natively, like build2 or others, as demonstrated by Phil Nash in this video.

Support for Makefile projects: How-to

Modern C++ tooling doesn’t have a single agreed way to handle Makefile projects in an IDE. We’ve identified several approaches (used across other IDEs and editors), which are outlined below:

Option 1: Compiler wrappers

There is a tool called scan-build, which helps you get a compilation database by intercepting compiler calls during the build. The first approach uses a similar concept – the IDE substitutes the actual compiler with wrappers (using CC and CXX environment variables, also through PATH) that would record the compilation command and then call an actual compiler.

The main benefit of such an approach is that it will work not only for Makefile projects, but for any build system. However, this requires a full clean build, which might take too long and might not be possible to do on the machine the IDE runs on.

Option 2: LD_PRELOAD

This approach also takes the idea from the currently existing tooling, which is Bear, and is similar to option 1. In a nutshell, on Unix-like systems, it is possible to set a LD_PRELOAD environment variable and specify a dynamic library that will be loaded before execution of any build process. This will allow intercepting any calls to the compiler.

This approach interferes with the build process less, which is important to some fragile configurations. But it’s Unix-specific (available also on macOS, but requires some special permissions).

Option 3: Parse the output of Make

The Make command prints lots of useful output during its work, which can be collected and reused for getting information about a project. This idea serves as a basis for the third approach. There is also a useful --just-print option, which helps avoid actually building the project during project reload, and so it’s possible to achieve better performance than a regular Make call.

This approach looks nice as it doesn’t affect the build process and allows us to collect the information quicker compared to the full project build. This is also a ‘portable’ option, as the IDE can theoretically start with the Make output recorded on another machine. So while this approach is not extendable to other build systems, it looks to us like the preferred solution from our early days of research.

Support for Makefile projects: CLion’s prototype

As we’ve mentioned above, instead of automating a workaround using a compilation database for Makefile projects by bringing the tools to intercept compiler calls to the user’s environment, we’ve decided to implement the approach of parsing Make’s output. While going this route, we had to deal with many interesting subtasks: distinguishing the compilation command from other shell commands, understanding working directories and their messed up output, and many other things that require some specific heuristics to be implemented. But after all, we’ve nailed it and got a working prototype of the Makefile projects analyzer inside CLion:

What could possibly go wrong?

Here is where it gets really exciting! The internal testing on a huge variety of Makefile projects gave us many hints on how to tune the heuristics. Let’s take a close look at the algorithm details to understand what could possibly go wrong.

CLion runs Make, reads its output, and tries to parse it in order to extract the compilation commands and working directory. Entering/Leaving directory <dir> messages in the output identify the working directory we are currently in. This information is required to understand which source file is actually being built, as file names are often specified relative to the working directory. In some projects these messages are also replaced with cd <directory> && gcc <file>. Accurately extracting this information is a crucial part of the algorithm.

It’s easy to fail here as there are widely used tricks to silence Make. Let’s dive deeper into the Make options! The default behavior for GNU Make is to print directories. Makefile can suppress it by using the .SILENT directive, but invoking Make with --print-directories overrides it. However, Makefile can override that by setting GNUMAKEFLAGS=--no-print-directory, which in turn could be overridden by passing GNUMAKEFLAGS=--print-directory as a command line option when invoking Make.

Inside a directory, the output messages are treated as potential compilation commands. CLion tries to parse them, looking for known compilers and their flags. In the cases where it fails, the line is considered to be just a text string and so is skipped. Interestingly, there are some wrappers like libtool which hide the compilation flags and interfere in the Make’s output, and so makes our current approach fail. Shell and linkage commands interfere as well, but it’s possible to teach the algorithm to skip them accurately.

Projects we used for testing

Let us list some of the projects we used for testing and point to some specifics we’ve found out while parsing the --just-print output. It’s worth mentioning that on most of them the approach that is currently prototyped in CLion does work:

1. Mbed TLS: https://github.com/ARMmbed/mbedtls.
   Comes with .SILENT target which disables printing directories.
2. A guide to basic Makefile projects used in Embedded Development: https://github.com/davepfeiffer/embedded-makefile-flow.
   Doesn’t print directories by default. But overall it’s very generic and simple from the perspective of parsing Make’s output.
3. GNU nano: https://git.savannah.gnu.org/cgit/nano.git.
   The project uses the Autotools, and the output includes many commands for dependency generation.
4. CPython: https://github.com/python/cpython.
   The project uses the Autotools. Several directories aren’t included in the target all (e.g. Modules).
5. OpenJDK: https://github.com/openjdk/jdk.
   This is a huge Autotools project with a customized build process, which hides almost all of the output. For now, our prototype fails on this project. However, we’ve found special make targets that are specifically responsible for IDE support:
   • compile-commands generates a compilation database which you can later open in CLion. (Later, we are planning to write a detailed blog post with instructions based on our internal usage of CLion for OpenJDK.)
   • vscode-project generates a VSCode project.
6. TscanCode static analysis tool: https://github.com/Tencent/TscanCode.
   It doesn’t print the directories, but overall it works fine with our approach.
7. Node.js: https://github.com/nodejs/node
   A huge project with lots of dependencies, which also uses a custom configure script. It’s Makefiles explicitly disable printing directories, but even without it the --just-print is enormously big.
8. Linux: https://github.com/torvalds/linux
   Custom build system (KBuild) is used here and normally it runs in a fairly quiet mode. However, changing to the verbose mode by passing V=1 helps. Still the --just-print output lines contain custom prefixes, and parsing the output produced without --just-print seems to work to some extent, but naturally takes a lot of time.
9. GCC: https://github.com/gcc-mirror/gcc
   Another huge Autotools project. More importantly, it uses a libtool wrapper and so hides some of the actual compilation flags.
10. And there are many others.

Call for help!

Obviously, the variety of Makefile projects is huge and you probably have some interesting projects in mind! We’ll be really happy to take a look and try them out with our prototyped solution. So please do share links to them in the comments.

And if you are ready to play with it on your own Makefile project that’s not generally available, please ping us (here in comments or drop a message to anastasia.kazakova at jetbrains.com) and we’ll share the EAP build and the instructions with you in private, so that you can give it a try and give us your feedback.

Your CLion team
JetBrains
The Drive to Develop

Subscribe to CLion Blog updates

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!