IntelliJ Scala plugin blog

For several years we’ve been working with Triplequote to help them develop the Hydra IntelliJ IDEA plugin. By combining IntelliJ IDEA with the Hydra parallel compiler, you can speed up both Scala development and Scala compilation. Here is a tutorial from the Triplequote team to explain the details.

1. What is Hydra?
2. Install the Hydra IntelliJ IDEA plugin
3. Compile!
4. Low memory detection
5. Compilation bottlenecks detection
6. What next?

What is Hydra?

Hydra, developed by Triplequote, is the only Scala compiler that parallelizes compilation across all available compute cores, speeding up compilation time up to 5x, and providing much faster return-to-productivity.

In addition to faster return to productivity, Hydra ships with built-in compile time monitoring, allowing you to crack open the compiler “black box” with the Hydra Dashboard, providing visualizations and metrics for optimizing your project’s compilation speed over time, and helping teams keep performance under control.

Hydra supports the full language (macros and compiler plugins – Scala.JS included), and it smoothly integrates with your build tool of choice, whether that is sbt, Maven or Gradle.

This article shows how to set up Hydra in IntelliJ IDEA and the many benefits delivered by Hydra in addition to parallel compilation.

Install the Hydra IntelliJ IDEA plugin

To install the Hydra IntelliJ IDEA plugin just search for “Triplequote Hydra” on the Plugins Marketplace.

Once installed, open the “Hydra Compiler” page and notice that at the top it says “No license detected”.

This is normal, as you need a license to compile with Hydra. Go ahead and hit the button “Get trial license” located on the top-right ([1]), and provide your full name and email. You will immediately receive the license key (check your SPAM folder if you can’t see it).

To use it, go back to the “Hydra Compiler” page, and click the “Enter license key” button ([2]). Paste the Hydra Developer license key you have received and proceed. The “Hydra Compiler” page is updated to confirm the validity of the license.

Congratulations! You are all set and ready to compile your Scala projects with Hydra.

Compile!

All Scala projects opened in IntelliJ IDEA will now be compiled with Hydra. You don’t need to do anything, it just works!

To try it out, hit “Build > Rebuild project” and notice in the Message view that “N Hydra workers” were used to compile your Scala sources.

Keep in mind that the default number of Hydra workers used may vary depending on the machine’s hardware, what type of Hydra license is in use, and the number of compiled sources. You can always modify it yourself on the settings page.

Low memory detection

A common (and often ignored) reason for long compilation time is that not enough memory heap is reserved for compilation. In fact, Scala compilation is allocation hungry and can put considerable pressure on the JVM Garbage Collector (GC), to the extreme that more time may be spent GC-ing rather than compiling!

The default heap size reserved for Scala compilation inside IntelliJ IDEA is 1024MB (see the “Scala Compiler Server” preference page under “Preferences” > “Build, Execution, Deployment” > “Compiler” > “Scala Compiler” > “Scala Compile Server”). While this default is good to get started, as modules grows in size and complexity more memory is often needed. Hence, memory usage needs to be constantly monitored during compilation, as otherwise it’s easy to miss that a reason why compilation takes long is because a sizeable amount of time is wasted GC-ing.

The solution to this rather annoying problem is to monitor GC. This can be achieved with the help of a profiler (e.g., VisualVM), but we prefer to automate and when using Hydra to compile there is no need to profile GC, ever. In the event that a considerable amount of time is spent GC-ing during compilation, Hydra reports a warning so that you can act on it.

To fix the problem, increase the max heap size in the “Scala Compiler Server” preference page. Be aware that for the change to be picked up the Compiler Server process needs to be restarted – click the throttle icon located at the bottom of the editor, then “Stop” and “Run” again.

Compilation bottlenecks detection

We, as Scala developers, have a tendency to accept our project’s compilation time without ever questioning it. After all, what could we possibly do about it? It turns out there is actually a lot we can do, but to direct our efforts we need to gain visibility into what the compiler does and know where the biggest bangs for the bucks are. Said otherwise, we need to know what Scala sources in our project takes the longest to compile.

One of the greatest features of Hydra is its built-in compile-time monitoring capabilities, and once again Hydra does the heavy lifting for you: if it detects a source file takes more than 5 seconds to compile, it reports the information at the end of the compilation.

It’s surprising how having this simple piece of information changes entirely the attitude we have in the face of the compiler. All of a sudden, we are surprised that some sources may be taking 5, or 10 seconds, or even more to compile, and hence we dig deeper. How do we do that? And how to know what source(s) takes long to compile? We look at the compilation data collected in the Hydra Dashboard!

The compilation metrics above were collected while compiling the Doobie open-source project. The Dashboard points out immediately what source took more than 5 seconds to compile.

This source file contains 250+ case classes definitions. For each case class, the Scala compiler generates several methods in both the class and its companion object. So, while a single case class is very fast to compile, the time adds up when you have over 250+ defined in a single source file. But using case classes was a design decision for the library, so in this case we are ok with the implied compile time cost. This is a remarkable insight we just gained on the project!

Despite the above considerations, there is still value in breaking up a source file that takes a long time to compile, as this reduces the incremental compilation time when editing the specific source. Thanks to IntelliJ IDEA refactoring capabilities, this is easy to achieve, as we can use the “Move” refactoring to split the logic into multiple files, bringing the compilation time of all sources below 5 seconds (the smaller the better!).

As you start exploring the compilation metrics collected on your project, you’ll discover bottlenecks you didn’t expect. A common one is to have imports in scope that are used to generate implicit values through macros (for instance, using the Circe automatic derivation feature). To learn more on this specific topic and how to address it, you may want to watch this ScalaDays talk “5 Things you need to know about Scala compilation” or read the Zalando article “Achieving 3.2x faster compilation time”. The Hydra Dashboard documentation also provides additional useful advice for speeding up compilation further.

What next?

Check out the Hydra documentation to find out more about Hydra and discover how easy it is to integrate it with your build tool of choice. Also, if you would like to benchmark how Hydra performs against the vanilla Scala compiler, you might find the hydraBenchmark command very useful!

And don’t forget to install the Dashboard to quickly identify compilation bottlenecks affecting your Scala projects!

Some releases are about adding more features. In this release, we’ve focused on improving the existing features of the Scala plugin. These features are the bread and butter of Scala programmers, so we hope the improvements will make our day-to-day work much easier. Here goes:

1. Highlighting of for-comprehensions
2. Find Usages for implicits
3. Decompile to Java for Scala .class files
4. Customizable Scalafmt version
5. Separate HOCON plugin
6. Error highlighting improvements

Highlighting of for-comprehensions

For-comprehensions can sometimes be… incomprehensible. Being a syntactic sugar for composing foreach, map, flatMap, and filter / withFilter operations, for-comprehensions look very differently from the implied method calls. As a Scala programmer, you can probably desugar for-comprehensions in your mind’s eye and, as long as your code compiles, all is well.

The problem begins when there’s an error (yes, once in a while this does happen, you know). An error inside a for-comprehension is actually an error inside its desugared form. Highlighting such an error is tricky, because, in addition to the desugaring, this requires some kind of reverse transformation (“error sugaring”?).

In this release, we present a solution that makes errors in for-comprehensions more comprehensible:

We treat <- as a reference to the corresponding foreach / map / flatMap method, and we treat if as a reference to the filter / withFilter method.

This also lets you invoke GoTo, Quick Definition, and Quick Documentation actions in those spots. For example:

The binding of <- and if symbols to the method calls makes it possible to display implicit arguments:

If that looks a bit unusual, it’s due to the limitation of the language syntax – there’s no way to pass explicit arguments to implicit parameters in a for-comprehension (should we create a SIP?). However, this is consistent with the chosen schema, and is required for showing implicits-related errors:

On top of all that, we’ve improved the desugaring of for-comprehensions. So, if you’re curious about what is behind the curtain of syntactic sugar, you can press Alt + Enter and Desugar for comprehension (or you can rely on the Code | Desugar Scala code action).

Find Usages for implicits

Speaking of implicits, previously the implicits hints allowed you to see usages of implicits and go from usages to definitions, but you couldn’t do the reverse – go from definitions to usages, by invoking Find Usages. Well, now you can:

This capability relies on bytecode indices, and can also be used to locate other things that are hidden behind the syntactic sugar, and not present in the source code “as is”, including:

• apply / unapply method calls,
• Single Abstract Method (SAM) type instantiations,
• foreach / map / flatMap / filter / withFilter calls via a for-comprehension (this nicely complements the for-comprehenison highlighting).

The feature goes beyond the typical search. While you can emulate Find Usages by means of a full-text search (albeit imperfectly), you cannot rely on text to find something that is not there. Together with the View | Show Implicit Hints, this feature puts you in control of the ‘invisible’ things.

Decompile to Java for Scala .class files

Did you know that IntelliJ IDEA can decompile .class files to Java? Now this is also possible for compiled Scala files. Suppose you have a third-party class that does something, but you don’t know what – because there are no sources at hand:

That’s where the Decompile to Java may come in handy:

Customizable Scalafmt version

While we supported the scalafmt formatter last year, the Scala plugin could use only a single, bundled version of scalafmt. Now IntelliJ IDEA can automatically download and use whatever version of scalafmt you configure in .scalafmt.conf:

The feature relies on the scalafmt-dynamic module, and so needs an Internet connection. Other than that, this doesn’t require any additional action on your part – everything just works (or at least we hope it does – let us know if it doesn’t for you!).

By the way, we’re also constantly improving the build-in IntelliJ IDEA formatter too

Separate HOCON plugin

Hocon is a “Human-Optimized Config Object Notation” format used by many ‘enterprise’ Scala frameworks such as Akka or Play. Because of that, previous versions of the plugin bundled the support of HOCON together with the support of Scala. And yet not everyone who uses Scala needs HOCON.

In this release, we’ve extracted the HOCON support into a separate repository and a separate plugin:

If you do need HOCON, you don’t have to do anything – the HOCON plugin will be automatically installed with the Scala plugin update (and, subsequently, updated). But now you have more flexibility – if you don’t use HOCON, you can easily disable or remove the HOCON plugin in Settings | Plugins.

(Kudos to Roman Janusz for developing the HOCON plugin!)

Error highlighting improvements

While we’re constantly working to make error highlighting even better, sometimes we improve it a lot… like this time! Here are a few highlights:

• Partial unification and type variable unification in general.
• Constructor highlighting, and calls to private constructors.
• better-monadic-for compiler plugin: implicit0 feature.
• kind-projector: value level polymorphic lambdas.
• simulacrum: higher-arity type constructors.

Here’s an example to inspire you:

We’re also refactoring our code to make it cleaner and clearer, so if you’ve ever wanted to contribute to the Scala plugin, now is a better time than ever.

While we’ve tried hard to prevent and avoid bugs, absolute perfection is impossible to attain. We’re planning a bugfix release, so please report any issues to YouTrack so that we can fix them as soon as possible.

By the way, the IntelliJ Scala plugin now has an official Twitter account: @IntelliJScala – feel free to follow us for more news and updates.

Develop with Pleasure! No, Drive to Develop! Oh… never mind. Just enjoy!
(and, your feedback is welcome)

Sincerely,

The IntelliJ Scala plugin team