Combine IntelliJ IDEA with Hydra for the Fastest Scala Development Experience

For several years we’ve been working with Triplequote to help them develop the Hydra IntelliJ IDEA plugin. By combining IntelliJ IDEA with the Hydra parallel compiler, you can speed up both Scala development and Scala compilation. Here is a tutorial from the Triplequote team to explain the details.

1. What is Hydra?
2. Install the Hydra IntelliJ IDEA plugin
3. Compile!
4. Low memory detection
5. Compilation bottlenecks detection
6. What next?

What is Hydra?

Hydra, developed by Triplequote, is the only Scala compiler that parallelizes compilation across all available compute cores, speeding up compilation time up to 5x, and providing much faster return-to-productivity.

In addition to faster return to productivity, Hydra ships with built-in compile time monitoring, allowing you to crack open the compiler “black box” with the Hydra Dashboard, providing visualizations and metrics for optimizing your project’s compilation speed over time, and helping teams keep performance under control.

Hydra supports the full language (macros and compiler plugins – Scala.JS included), and it smoothly integrates with your build tool of choice, whether that is sbt, Maven or Gradle.

This article shows how to set up Hydra in IntelliJ IDEA and the many benefits delivered by Hydra in addition to parallel compilation.

Install the Hydra IntelliJ IDEA plugin

To install the Hydra IntelliJ IDEA plugin just search for “Triplequote Hydra” on the Plugins Marketplace.1-installation

Once installed, open the “Hydra Compiler” page and notice that at the top it says “No license detected”.2-license-missing

This is normal, as you need a license to compile with Hydra. Go ahead and hit the button “Get trial license” located on the top-right ([1]), and provide your full name and email. You will immediately receive the license key (check your SPAM folder if you can’t see it).

To use it, go back to the “Hydra Compiler” page, and click the “Enter license key” button ([2]). Paste the Hydra Developer license key you have received and proceed. The “Hydra Compiler” page is updated to confirm the validity of the license.3-license-valid

Congratulations! You are all set and ready to compile your Scala projects with Hydra.

Compile!

All Scala projects opened in IntelliJ IDEA will now be compiled with Hydra. You don’t need to do anything, it just works!

To try it out, hit “Build > Rebuild project” and notice in the Message view that “N Hydra workers” were used to compile your Scala sources.4-compile-with-hydra

Keep in mind that the default number of Hydra workers used may vary depending on the machine’s hardware, what type of Hydra license is in use, and the number of compiled sources. You can always modify it yourself on the settings page.

Low memory detection

A common (and often ignored) reason for long compilation time is that not enough memory heap is reserved for compilation. In fact, Scala compilation is allocation hungry and can put considerable pressure on the JVM Garbage Collector (GC), to the extreme that more time may be spent GC-ing rather than compiling!

The default heap size reserved for Scala compilation inside IntelliJ IDEA is 1024MB (see the “Scala Compiler Server” preference page under “Preferences” > “Build, Execution, Deployment” > “Compiler” > “Scala Compiler” > “Scala Compile Server”). While this default is good to get started, as modules grows in size and complexity more memory is often needed. Hence, memory usage needs to be constantly monitored during compilation, as otherwise it’s easy to miss that a reason why compilation takes long is because a sizeable amount of time is wasted GC-ing.

The solution to this rather annoying problem is to monitor GC. This can be achieved with the help of a profiler (e.g., VisualVM), but we prefer to automate and when using Hydra to compile there is no need to profile GC, ever. In the event that a considerable amount of time is spent GC-ing during compilation, Hydra reports a warning so that you can act on it.5-gc-overhead

To fix the problem, increase the max heap size in the “Scala Compiler Server” preference page. Be aware that for the change to be picked up the Compiler Server process needs to be restarted – click the throttle icon located at the bottom of the editor, then “Stop” and “Run” again.
6-restart-compile-server

Compilation bottlenecks detection

We, as Scala developers, have a tendency to accept our project’s compilation time without ever questioning it. After all, what could we possibly do about it? It turns out there is actually a lot we can do, but to direct our efforts we need to gain visibility into what the compiler does and know where the biggest bangs for the bucks are. Said otherwise, we need to know what Scala sources in our project takes the longest to compile.

One of the greatest features of Hydra is its built-in compile-time monitoring capabilities, and once again Hydra does the heavy lifting for you: if it detects a source file takes more than 5 seconds to compile, it reports the information at the end of the compilation.7-bottleneck-detected

It’s surprising how having this simple piece of information changes entirely the attitude we have in the face of the compiler. All of a sudden, we are surprised that some sources may be taking 5, or 10 seconds, or even more to compile, and hence we dig deeper. How do we do that? And how to know what source(s) takes long to compile? We look at the compilation data collected in the Hydra Dashboard!8-hydra-dashboard

The compilation metrics above were collected while compiling the Doobie open-source project. The Dashboard points out immediately what source took more than 5 seconds to compile.9-slowest-source

This source file contains 250+ case classes definitions. For each case class, the Scala compiler generates several methods in both the class and its companion object. So, while a single case class is very fast to compile, the time adds up when you have over 250+ defined in a single source file. But using case classes was a design decision for the library, so in this case we are ok with the implied compile time cost. This is a remarkable insight we just gained on the project!

Despite the above considerations, there is still value in breaking up a source file that takes a long time to compile, as this reduces the incremental compilation time when editing the specific source. Thanks to IntelliJ IDEA refactoring capabilities, this is easy to achieve, as we can use the “Move” refactoring to split the logic into multiple files, bringing the compilation time of all sources below 5 seconds (the smaller the better!).

As you start exploring the compilation metrics collected on your project, you’ll discover bottlenecks you didn’t expect. A common one is to have imports in scope that are used to generate implicit values through macros (for instance, using the Circe automatic derivation feature). To learn more on this specific topic and how to address it, you may want to watch this ScalaDays talk “5 Things you need to know about Scala compilation” or read the Zalando article “Achieving 3.2x faster compilation time”. The Hydra Dashboard documentation also provides additional useful advice for speeding up compilation further.

What next?

Check out the Hydra documentation to find out more about Hydra and discover how easy it is to integrate it with your build tool of choice. Also, if you would like to benchmark how Hydra performs against the vanilla Scala compiler, you might find the hydraBenchmark command very useful!

And don’t forget to install the Dashboard to quickly identify compilation bottlenecks affecting your Scala projects!

About Pavel Fatin

Programming enthusiast, technology advocate. IntelliJ Scala plugin developer at JetBrains, https://pavelfatin.com
This entry was posted in Uncategorized. Bookmark the permalink.

11 Responses to Combine IntelliJ IDEA with Hydra for the Fastest Scala Development Experience

  1. Vladimir Parfinenko says:

    Am I right that to know what sources take long time to compile I have to install some Docker-stuff with client-server architecture (aka Dashboard)?

  2. roman says:

    Have a problem to running idea with the plugin.

    java.io.IOException: com.intellij.ide.plugins.PluginManager$StartupAbortedException: Fatal error initializing plugin com.triplequote.hydra
    at com.intellij.openapi.project.impl.ProjectManagerImpl.loadProjectWithProgress(ProjectManagerImpl.java:561)
    at com.intellij.openapi.project.impl.ProjectManagerImpl.convertAndLoadProject(ProjectManagerImpl.java:536)
    at com.intellij.platform.PlatformProjectOpenProcessor.doOpenProject(PlatformProjectOpenProcessor.java:218)
    at com.intellij.ide.RecentProjectsManagerBase.doOpenProject(RecentProjectsManagerBase.java:511)
    at com.intellij.ide.RecentProjectsManagerBase.doReopenLastProject(RecentProjectsManagerBase.java:633)
    at com.intellij.ide.RecentProjectsManagerBase$MyAppLifecycleListener.appStarting(RecentProjectsManagerBase.java:691)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)

  3. Greg Silin says:

    Thanks for making this available!

    I’m using Custom SBT (not bundled), pointing to v 1.2.8 installed on my machine, and I have the setting “Use SBT shell” enabled for builds and imports. Is that set up compatible with Hydra? Do I need to disable building via SBT for Hydra to take over?

    • Mirco Dotta says:

      Hi Greg,

      Sure thing your setup is compatible. Though, you need to add the sbt-hydra plugin to your build definition, as you are delegating compilation to sbt. Please, click on the trial page on the Triplequote website and follow the sbt instructions (I’m not adding explicit links because otherwise my reply would be delayed because of moderation – feel free to get in touch at support[at]triplequote[dot]com if you can’t find the doc).

      • Greg Silin says:

        Duh, obviously :) was hungup on it being an IntelliJ plugin. Thanks so much for a quick reply!

        Looks like SBT plugin doesn’t require a license like the IntelliJ one?

        • Mirco Dotta says:

          No worries!

          The trial license is needed in both cases, but the process to obtain it is slightly different, as it is adapted to the tool you are using to get started with Hydra. Of course, you only need to go through it once, regardless of the development tool you choose to get started with.

  4. Pavel, you wrote “speeding up compilation time up to 5x, and providing orders of magnitude faster return-to-productivity.”

    A 5x improvement would be only 50% of one order of magnitude. A 10x improvement would be one order of magnitude. A 100x improvement would be 2 orders of magnitude. “Orders of magnitude” means that the improvement is in excess of 100x. This is clearly untrue. Please correct the blog.

    • Pavel Fatin says:

      Hi Michael,

      That’s a guest post from the Triplequote team.

      I guess that the “orders of magnitude” thing is supposed to be metaphorical, but, strictly speaking, the wording is not technically correct. I’ve replaced the “orders of magnitude faster” with “much faster” :)

      Thanks for catching that!

Leave a Reply

Your email address will not be published. Required fields are marked *