Dealing with Makefile Projects in CLion: Status Update

Posted on by Anastasia Kazakova

Update: Makefile projects support is now public in CLion 2020.2.

What request in our tracker has more than 1000 votes, 370 comments, and 800 watchers? You guessed it: Support Makefile projects. This has been a story of interesting findings, semi-automatic workarounds, and a long battle that we still plan to win! If you’d like to get the latest news on this topic, please read on.

From past to present: the evolution of project models in CLion

As you know, all work in CLion is done within the context of a project. Projects serve as the basis for coding assistance, bulk refactoring, coding style consistency, and other smart features. To understand the project model, CLion collects not only the list of project files, but also compilation flags, header search paths, and project-model specific variables. The IDE can create several resolve configurations aggregating these parameters, which you can switch in the editor when necessary.

5 years ago CLion was launched with CMake-only projects, and we shared the reasoning behind that decision in our blog. Since then, CMake has been growing steadily in the C++ community and finally overtook Visual Studio to become the most popular project model / build system for C++ development.

However, in the world of C++ there is no single standard project model. There are Makefile projects, qmake, msbuild, Bazel, Scons, and many others. Makefile projects are in the top 3, popular in OSS and embedded projects, and widely used for cross-platform C++ development, which makes them the best candidate to add to CLion.

On our journey to supporting Makefile projects, we’ve taken the following steps:

  • Open a folder in CLion. With no project model in place, CLion treats all the code on a very basic level and can’t provide any smart coding assistance (like Go to Symbol or refactorings).
  • Compilation database support. This is a truly universal project format, which can be generated from any project model, including any custom one. CLion provides full coding assistance for such projects, and often that’s the easiest way to open an arbitrary project in CLion.
  • Support for custom build targets and custom Run/Debug configurations. We added this to enrich the compilation database or folder-based projects experience, as build/clean commands can then be used by CLion and such applications can be run and debugged.

While it’s now possible to build, run and debug any projects (including Makefile-based) in CLion by using Custom Build Targets, it requires an accurate manual setup. The CMake project experience is much better here – CLion automatically detects available targets to build, and executables to run/debug. We’re inclined to provide this experience for Makefile-based projects as well, however, this requires a lot of heuristics and tuning. In the meantime, you can use existing Makefile plugin to run make targets (note that it doesn’t allow running or debugging executables, you still should use Custom Build Targets for this).

This brought us to the workflow where users work around Makefile projects with the help of File Watchers and the compilation database. The main pain-point is that you have to install the tools to extract the compilation database from your Makefiles. However, the workflow is universal and works smoothly for any project model not supported natively, like build2 or others, as demonstrated by Phil Nash in this video.

Support for Makefile projects: How-to

Modern C++ tooling doesn’t have a single agreed way to handle Makefile projects in an IDE. We’ve identified several approaches (used across other IDEs and editors), which are outlined below:

Option 1: Compiler wrappers

There is a tool called scan-build, which helps you get a compilation database by intercepting compiler calls during the build. The first approach uses a similar concept – the IDE substitutes the actual compiler with wrappers (using CC and CXX environment variables, also through PATH) that would record the compilation command and then call an actual compiler.

The main benefit of such an approach is that it will work not only for Makefile projects, but for any build system. However, this requires a full clean build, which might take too long and might not be possible to do on the machine the IDE runs on.

Option 2: LD_PRELOAD

This approach also takes the idea from the currently existing tooling, which is Bear, and is similar to option 1. In a nutshell, on Unix-like systems, it is possible to set a LD_PRELOAD environment variable and specify a dynamic library that will be loaded before execution of any build process. This will allow intercepting any calls to the compiler.

This approach interferes with the build process less, which is important to some fragile configurations. But it’s Unix-specific (available also on macOS, but requires some special permissions).

Option 3: Parse the output of Make

The Make command prints lots of useful output during its work, which can be collected and reused for getting information about a project. This idea serves as a basis for the third approach. There is also a useful --just-print option, which helps avoid actually building the project during project reload, and so it’s possible to achieve better performance than a regular Make call.

This approach looks nice as it doesn’t affect the build process and allows us to collect the information quicker compared to the full project build. This is also a ‘portable’ option, as the IDE can theoretically start with the Make output recorded on another machine. So while this approach is not extendable to other build systems, it looks to us like the preferred solution from our early days of research.

Support for Makefile projects: CLion’s prototype

As we’ve mentioned above, instead of automating a workaround using a compilation database for Makefile projects by bringing the tools to intercept compiler calls to the user’s environment, we’ve decided to implement the approach of parsing Make’s output. While going this route, we had to deal with many interesting subtasks: distinguishing the compilation command from other shell commands, understanding working directories and their messed up output, and many other things that require some specific heuristics to be implemented. But after all, we’ve nailed it and got a working prototype of the Makefile projects analyzer inside CLion:
Makefiles

What could possibly go wrong?

Here is where it gets really exciting! The internal testing on a huge variety of Makefile projects gave us many hints on how to tune the heuristics. Let’s take a close look at the algorithm details to understand what could possibly go wrong.

CLion runs Make, reads its output, and tries to parse it in order to extract the compilation commands and working directory. Entering/Leaving directory <dir> messages in the output identify the working directory we are currently in. This information is required to understand which source file is actually being built, as file names are often specified relative to the working directory. In some projects these messages are also replaced with cd <directory> && gcc <file>. Accurately extracting this information is a crucial part of the algorithm.

It’s easy to fail here as there are widely used tricks to silence Make. Let’s dive deeper into the Make options! The default behavior for GNU Make is to print directories. Makefile can suppress it by using the .SILENT directive, but invoking Make with --print-directories overrides it. However, Makefile can override that by setting GNUMAKEFLAGS=--no-print-directory, which in turn could be overridden by passing GNUMAKEFLAGS=--print-directory as a command line option when invoking Make.

Inside a directory, the output messages are treated as potential compilation commands. CLion tries to parse them, looking for known compilers and their flags. In the cases where it fails, the line is considered to be just a text string and so is skipped. Interestingly, there are some wrappers like libtool which hide the compilation flags and interfere in the Make’s output, and so makes our current approach fail. Shell and linkage commands interfere as well, but it’s possible to teach the algorithm to skip them accurately.

Projects we used for testing

Let us list some of the projects we used for testing and point to some specifics we’ve found out while parsing the --just-print output. It’s worth mentioning that on most of them the approach that is currently prototyped in CLion does work:

  1. Mbed TLS: https://github.com/ARMmbed/mbedtls.
    Comes with .SILENT target which disables printing directories.
  2. A guide to basic Makefile projects used in Embedded Development: https://github.com/davepfeiffer/embedded-makefile-flow.
    Doesn’t print directories by default. But overall it’s very generic and simple from the perspective of parsing Make’s output.
  3. GNU nano: https://git.savannah.gnu.org/cgit/nano.git.
    The project uses the Autotools, and the output includes many commands for dependency generation.
  4. CPython: https://github.com/python/cpython.
    The project uses the Autotools. Several directories aren’t included in the target all (e.g. Modules).
  5. OpenJDK: https://github.com/openjdk/jdk.
    This is a huge Autotools project with a customized build process, which hides almost all of the output. For now, our prototype fails on this project. However, we’ve found special make targets that are specifically responsible for IDE support:

    • compile-commands generates a compilation database which you can later open in CLion. (Later, we are planning to write a detailed blog post with instructions based on our internal usage of CLion for OpenJDK.)
    • vscode-project generates a VSCode project.
  6. TscanCode static analysis tool: https://github.com/Tencent/TscanCode.
    It doesn’t print the directories, but overall it works fine with our approach.
  7. Node.js: https://github.com/nodejs/node
    A huge project with lots of dependencies, which also uses a custom configure script. It’s Makefiles explicitly disable printing directories, but even without it the --just-print is enormously big.
  8. Linux: https://github.com/torvalds/linux
    Custom build system (KBuild) is used here and normally it runs in a fairly quiet mode. However, changing to the verbose mode by passing V=1 helps. Still the --just-print output lines contain custom prefixes, and parsing the output produced without --just-print seems to work to some extent, but naturally takes a lot of time.
  9. GCC: https://github.com/gcc-mirror/gcc
    Another huge Autotools project. More importantly, it uses a libtool wrapper and so hides some of the actual compilation flags.
  10. And there are many others.

Call for help!

Obviously, the variety of Makefile projects is huge and you probably have some interesting projects in mind! We’ll be really happy to take a look and try them out with our prototyped solution. So please do share links to them in the comments.

And if you are ready to play with it on your own Makefile project that’s not generally available, please ping us (here in comments or drop a message to anastasia.kazakova at jetbrains.com) and we’ll share the EAP build and the instructions with you in private, so that you can give it a try and give us your feedback.

Your CLion team
JetBrains
The Drive to Develop

Comments below can no longer be edited.

51 Responses to Dealing with Makefile Projects in CLion: Status Update

  1. Marco Cecchetti says:

    February 18, 2020

    Hi Анастасия,
    I’m a LibreOffice developer and also a software engineer at Collabora which provides a cloud version of LibreOffice. I’d very like to see the new CLion’s prototype tested on the LibreOffice project. I know that’s a huge project (a full build on a 4C/4T machine requires 2 hours), however I think you could get many useful hints from testing it.
    Usually I config it with ./autogen.sh --enable-dbgutil --enable-werror and build it with make build-nocheck (which skips the unit tests and save 30min).
    Anyway here you can find more details on how to build LibreOffice: https://wiki.documentfoundation.org/Development/BuildingOnLinux

    I think that’s really a great news that you are working hard on supporting makefile projects as first class citizen in CLion!
    I’m not a make/makefile expert, anyway I’d like to help in getting LibreOffice working with the new solution you are implementing!

    All the best,
    — Marco

    • Anastasia Kazakova says:

      February 18, 2020

      Thanks, Marco. We’ll give it a try and will definitely use your tips.

  2. Zhu Lingshan says:

    February 18, 2020

    I am a kernel developer, I hope CLion can handle kernel source code. Here is a guide on how to play with kernel:
    1. git clone https://github.com/torvalds/linux.git
    2. type “make menuconfig” and select “save”. You will get a .config file in the source code folder, which is basically a bunch of switches controlling a feature would be built in kernel(Y), built as a module(M) or not built(N). Please note, Kconfig relies on them.
    3. type make -j4 (4 jobs make).

    There is a top level makefile and every sub-folder has its own makefile, you may also find Kconfig there.

    Kernel is a huge project, so I guess it may be a challenge in indexing the source code. Last year I tried CLion, it freezes….

    I really appreciate your and the team’s hard working on this feature. Please send me an email if you have any troubles understanding the kernel.

    • Anastasia Kazakova says:

      February 19, 2020

      Thank you for your feedback. As you can see from the blog post we managed to deal with the Linux kernel:

      Custom build system (KBuild) is used here and normally it runs in a fairly quiet mode. However, changing to the verbose mode by passing V=1 helps. Still the –just-print output lines contain custom prefixes, and parsing the output produced without –just-print seems to work to some extent, but naturally takes a lot of time.

  3. Sean Johnston says:

    February 19, 2020

    PostgreSQL uses make (gnu make specifically) and I didn’t notice it listed explicitly as a test case. It also recursively calls make on sub directories. Be interesting to see how that gets handled.

    • Anastasia Kazakova says:

      February 19, 2020

      Thank you. We’ve put it to our test cases list. It’s definitely longer than the one I’ve shared, but seems we missed PostgreSQL there.

  4. Alex Richardson says:

    February 21, 2020

    QEMU might also be an interesting project to test.
    I’ve tried generating a compilation DB with compiledb, but that only partially worked since it didn’t get the directories right.
    QEMU uses a GNU makefile and custom configure script.
    To build you run ./configure and then make but be sure to pass V=1 to get the commands printed.

    If you are looking for a really challenging makefile project:
    FreeBSD is a very large make-based project (it includes all of LLVM). This will be quite difficult since it uses bmake and not GNU make, so flags such as –just-print aren’t accepted.
    My current workaround is to have a local CMakeLists.txt that passes the right -I flags to get reasonable code completion (at least for the kernel bits).

    • Anastasia Kazakova says:

      February 21, 2020

      Thanks. Both are on our internal list, but we haven’t got to them yet. Will definitely try!

  5. Maxim Yanchenko says:

    February 21, 2020

    Hi Anastasia,

    Please correct me if I’m wrong but it currently looks like there is just 1 compile_commands.json for a project.
    I believe this is wrong because there can be many mode of building a project (either with targets or with env vars), so it should really be a compile_commands.json per configuration.
    E.g. threaded/non-threaded, debug/release etc can actually have different files built and included, not to mention different flags and active defines. Some flags can be passed from the environment etc, like USE_QT=1 vs. USE_AGG=1 etc that would affect what you actually include and link.
    If you have configuration first, then you can have separate compile dbs and natural build/clean variants.

    The model “one compile_commands.json to rule them all” doesn’t really work with real projects.

    • Anastasia Kazakova says:

      February 21, 2020

      I somehow agree that it looks reasonable. Thanks.

  6. Marc Rosen says:

    February 21, 2020

    Hi,
    how about testing it against NuttX RTOS (https://www.nuttx.org)?
    This MCU OS uses KBuild to manage its configuration for different drivers, archs and boards and make for building.

    The source code is available at https://github.com/apache/incubator-nuttx and https://github.com/apache/incubator-nuttx-apps. A standalone distribution of kfrontend is available at https://bitbucket.org/nuttx/tools.

    • Anastasia Kazakova says:

      February 21, 2020

      Thanks, added to our list.

  7. James Fuller says:

    February 21, 2020

    https://github.com/curl/curl please

    • Anastasia Kazakova says:

      February 22, 2020

      Ok, we’ll check. Though it has also CMake model.

  8. Tuomas Tynkkynen says:

    February 22, 2020

    Hi, I wonder if it has been considered that instead of parsing the output, GNU make could be modified to write the commands it is executing to some place in a machine-readable format (for example, modify make to look for some environment variable, which if set should be a path to a UNIX domain socket where the executed commands are listed).

    Of course, it will be annoying since you would need to ship a prebuilt custom GNU make with CLion, but maybe eventually such patch could be accepted into upstream GNU make.

    Regardless, great to hear progress on supporting this!

    • Anastasia Kazakova says:

      February 22, 2020

      Thank you. I don’t think shipping and working only with some custom Make version will be a good option for general Makefiles users in CLion. Actually, wrapping compilers might look somehow similar to what you suggest, with all the pros and cons discussed there.

  9. Jason says:

    February 29, 2020

    Tested Clion EAP failed to make with Gnuplot (https://github.com/gnuplot/gnuplot.git). In one case it seemed to succeed but all source files are marked as “not part of the project”. Also the IDE should recognize “makefile”, currently only recognizes “Makefile”

  10. Jason says:

    February 29, 2020

    How do I open a project “root” which contains subdir “Debug” where the makefile resides ? This is typical structure for Eclipse CDT managed builds.
    Open directory “root” -> did nothing
    Open directory “Debug” -> did nothing
    Open a file Debug>Makefile -> worked. But project view loses files hierarchy of “root”.

  11. Michael Cummings says:

    March 4, 2020

    A large project you might try is PHP. It has a shall we say interesting build scripting mix. Newer ones are much better but can still be very challenging on windows especially

    • Anastasia Kazakova says:

      March 4, 2020

      Thanks, we have PHP on our radar to try with.

  12. Mikael Lagerkvist says:

    March 4, 2020

    I would appreciate if you could test the Gecode project (https://www.gecode.org). There exists a CMake solution, but the official way to compile is using configure and make.

    The code can be downloaded from https://github.com/Gecode/gecode and the compilation instructions are available in the manual https://www.gecode.org/doc-latest/MPG.pdf in section 2.6.2.

    In particular, it might be interesting to test
    * If detection of and compilation with the Qt libraries works.
    * How to detect and set the different configure options.
    * Setting options for and compiling with GMP and MPFR.
    * Configuring and compiling in a different directory than the source root.
    * Running tests (make check for a small test set)

    Please don’t hesitate to reach out to me if you have any questions if you actually try it.

    • Anastasia Kazakova says:

      March 4, 2020

      Thanks. Do you want to try on your own as well?

      • Mikael Lagerkvist says:

        March 5, 2020

        I’d be happy to try it on my own as well.

        • Anastasia Kazakova says:

          March 6, 2020

          Which email address I should use to contact you?

          • Mikael Lagerkvist says:

            March 9, 2020

            You can Contact me at user zayenz on gmail.

            • Anastasia Kazakova says:

              March 10, 2020

              Sent you an email

  13. Julian says:

    March 10, 2020

    Please check out https://github.com/ChibiOS/ChibiOS. CLion has a nice openOCD support by now and it’s a lot nicer to code in that Eclipse (imho! 🙂 )

    I’m currently using a CMake file that executes make and creates a compilation database, but thats slow and tiresome to use.

    Maybe you/some plugin even could implement some of the features ChibiStudio (eclipse based) provides.

    • Anastasia Kazakova says:

      March 10, 2020

      Thanks, we’ll take a look

      • Julian says:

        March 11, 2020

        Is there an ETA for a beta or EAP version for those functionalities?

        • Anastasia Kazakova says:

          March 11, 2020

          The plugin is now prototyped and shared privately with those who’d like to play with it. We see some major issues and directions for further improvements, so definitely won’t release it as a public functionality in the 2020.1. But are considering the public preview along with 2020.1 release or slightly later.

          • Julian says:

            March 23, 2020

            Could you send me the EAP Build so I can play with it? 🙂

            • Anastasia Kazakova says:

              March 23, 2020

              Should I use the email from here?

          • Julian Schneider says:

            March 25, 2020

            Yes, that should be fine, thank you!

          • Julian Schneider says:

            March 25, 2020

            Yes please use my email from here.

          • Martin says:

            April 21, 2020

            Any chance to have a look at the prototype?

            Feel free to reply via email to my email here.

            • Anastasia Kazakova says:

              April 21, 2020

              Will do

          • Rafal says:

            May 18, 2020

            Could you send to me this prototype? Please use my email.

            • Anastasia Kazakova says:

              May 18, 2020

              Sent you an email

  14. harryhare says:

    March 20, 2020

    redis:
    https://github.com/antirez/redis.git

    • Anastasia Kazakova says:

      March 20, 2020

      thanks

  15. Omar says:

    April 17, 2020

    Contiki-ng

    https://github.com/contiki-ng/contiki-ng

    I’m also interested in trying it out.

    • Anastasia Kazakova says:

      April 17, 2020

      Can I contact you via the email from the blog? Or ping me on anastasia.kazakova at jetbrains.com

  16. tolyan says:

    May 28, 2020

    glibc could be a good benchmark/test

    • Anastasia Kazakova says:

      May 29, 2020

      Thanks. Yes. We have it on the list.

  17. Dan says:

    June 1, 2020

    This is something I’d really like CLion to support (and with remote projects too please). No suggestions for a large project except for a large commercial in-house one which is of course is impossible for you to test, but have you considered this way of dumping makefile variables which may be more reliable than parsing the printed output?

    https://www.cmcrossroads.com/article/dumping-every-makefile-variable

    There is no need to use this method exclusively, you could dump variables and also parse printed output. But it seems to me that dumping variables is less hit-and-miss than parsing output.

    • Semyon Kolton says:

      June 1, 2020

      Hi, thanks for an interesting link. We actually considered using very similar technique, to extract some makefiles information. Unfortunately it’s also very fragile. So we decided for the time being to focus on parsing output. But most likely in the future we will also utilize this technique. Probably they will work in combination, but I’m not sure for now how.
      Regarding remote support, at the start Makefile projects will not support remote, but we hopefully will focus on it in the nearest future.

  18. Russell Dunk says:

    June 11, 2020

    Dungeon Crawl! https://github.com/crawl/crawl
    It’s a open source rogue-like game.
    Currently it’s having trouble with stuff that the Makefile kicks off before getting into the actual compilation of the game.

Subscribe

Subscribe for updates