Combine IntelliJ IDEA with Hydra for the Fastest Scala Development Experience

For several years we’ve been working with Triplequote to help them develop the Hydra IntelliJ IDEA plugin. By combining IntelliJ IDEA with the Hydra parallel compiler, you can speed up both Scala development and Scala compilation. Here is a tutorial from the Triplequote team to explain the details.

1. What is Hydra?
2. Install the Hydra IntelliJ IDEA plugin
3. Compile!
4. Low memory detection
5. Compilation bottlenecks detection
6. What next?

What is Hydra?

Hydra, developed by Triplequote, is the only Scala compiler that parallelizes compilation across all available compute cores, speeding up compilation time up to 5x, and providing much faster return-to-productivity.

In addition to faster return to productivity, Hydra ships with built-in compile time monitoring, allowing you to crack open the compiler “black box” with the Hydra Dashboard, providing visualizations and metrics for optimizing your project’s compilation speed over time, and helping teams keep performance under control.

Hydra supports the full language (macros and compiler plugins – Scala.JS included), and it smoothly integrates with your build tool of choice, whether that is sbt, Maven or Gradle.

This article shows how to set up Hydra in IntelliJ IDEA and the many benefits delivered by Hydra in addition to parallel compilation.

Install the Hydra IntelliJ IDEA plugin

To install the Hydra IntelliJ IDEA plugin just search for “Triplequote Hydra” on the Plugins Marketplace.1-installation

Once installed, open the “Hydra Compiler” page and notice that at the top it says “No license detected”.2-license-missing

This is normal, as you need a license to compile with Hydra. Go ahead and hit the button “Get trial license” located on the top-right ([1]), and provide your full name and email. You will immediately receive the license key (check your SPAM folder if you can’t see it).

To use it, go back to the “Hydra Compiler” page, and click the “Enter license key” button ([2]). Paste the Hydra Developer license key you have received and proceed. The “Hydra Compiler” page is updated to confirm the validity of the license.3-license-valid

Congratulations! You are all set and ready to compile your Scala projects with Hydra.

Compile!

All Scala projects opened in IntelliJ IDEA will now be compiled with Hydra. You don’t need to do anything, it just works!

To try it out, hit “Build > Rebuild project” and notice in the Message view that “N Hydra workers” were used to compile your Scala sources.4-compile-with-hydra

Keep in mind that the default number of Hydra workers used may vary depending on the machine’s hardware, what type of Hydra license is in use, and the number of compiled sources. You can always modify it yourself on the settings page.

Low memory detection

A common (and often ignored) reason for long compilation time is that not enough memory heap is reserved for compilation. In fact, Scala compilation is allocation hungry and can put considerable pressure on the JVM Garbage Collector (GC), to the extreme that more time may be spent GC-ing rather than compiling!

The default heap size reserved for Scala compilation inside IntelliJ IDEA is 1024MB (see the “Scala Compiler Server” preference page under “Preferences” > “Build, Execution, Deployment” > “Compiler” > “Scala Compiler” > “Scala Compile Server”). While this default is good to get started, as modules grows in size and complexity more memory is often needed. Hence, memory usage needs to be constantly monitored during compilation, as otherwise it’s easy to miss that a reason why compilation takes long is because a sizeable amount of time is wasted GC-ing.

The solution to this rather annoying problem is to monitor GC. This can be achieved with the help of a profiler (e.g., VisualVM), but we prefer to automate and when using Hydra to compile there is no need to profile GC, ever. In the event that a considerable amount of time is spent GC-ing during compilation, Hydra reports a warning so that you can act on it.5-gc-overhead

To fix the problem, increase the max heap size in the “Scala Compiler Server” preference page. Be aware that for the change to be picked up the Compiler Server process needs to be restarted – click the throttle icon located at the bottom of the editor, then “Stop” and “Run” again.
6-restart-compile-server

Compilation bottlenecks detection

We, as Scala developers, have a tendency to accept our project’s compilation time without ever questioning it. After all, what could we possibly do about it? It turns out there is actually a lot we can do, but to direct our efforts we need to gain visibility into what the compiler does and know where the biggest bangs for the bucks are. Said otherwise, we need to know what Scala sources in our project takes the longest to compile.

One of the greatest features of Hydra is its built-in compile-time monitoring capabilities, and once again Hydra does the heavy lifting for you: if it detects a source file takes more than 5 seconds to compile, it reports the information at the end of the compilation.7-bottleneck-detected

It’s surprising how having this simple piece of information changes entirely the attitude we have in the face of the compiler. All of a sudden, we are surprised that some sources may be taking 5, or 10 seconds, or even more to compile, and hence we dig deeper. How do we do that? And how to know what source(s) takes long to compile? We look at the compilation data collected in the Hydra Dashboard!8-hydra-dashboard

The compilation metrics above were collected while compiling the Doobie open-source project. The Dashboard points out immediately what source took more than 5 seconds to compile.9-slowest-source

This source file contains 250+ case classes definitions. For each case class, the Scala compiler generates several methods in both the class and its companion object. So, while a single case class is very fast to compile, the time adds up when you have over 250+ defined in a single source file. But using case classes was a design decision for the library, so in this case we are ok with the implied compile time cost. This is a remarkable insight we just gained on the project!

Despite the above considerations, there is still value in breaking up a source file that takes a long time to compile, as this reduces the incremental compilation time when editing the specific source. Thanks to IntelliJ IDEA refactoring capabilities, this is easy to achieve, as we can use the “Move” refactoring to split the logic into multiple files, bringing the compilation time of all sources below 5 seconds (the smaller the better!).

As you start exploring the compilation metrics collected on your project, you’ll discover bottlenecks you didn’t expect. A common one is to have imports in scope that are used to generate implicit values through macros (for instance, using the Circe automatic derivation feature). To learn more on this specific topic and how to address it, you may want to watch this ScalaDays talk “5 Things you need to know about Scala compilation” or read the Zalando article “Achieving 3.2x faster compilation time”. The Hydra Dashboard documentation also provides additional useful advice for speeding up compilation further.

What next?

Check out the Hydra documentation to find out more about Hydra and discover how easy it is to integrate it with your build tool of choice. Also, if you would like to benchmark how Hydra performs against the vanilla Scala compiler, you might find the hydraBenchmark command very useful!

And don’t forget to install the Dashboard to quickly identify compilation bottlenecks affecting your Scala projects!

Posted in Uncategorized | 11 Comments

IntelliJ Scala Plugin 2019.1: Highlighting For-comprehensions, Find Usages for Implicits, and More

Some releases are about adding more features. In this release, we’ve focused on improving the existing features of the Scala plugin. These features are the bread and butter of Scala programmers, so we hope the improvements will make our day-to-day work much easier. Here goes:

  1. Highlighting of for-comprehensions
  2. Find Usages for implicits
  3. Decompile to Java for Scala .class files
  4. Customizable Scalafmt version
  5. Separate HOCON plugin
  6. Error highlighting improvements

Highlighting of for-comprehensions

For-comprehensions can sometimes be… incomprehensible. Being a syntactic sugar for composing foreach, map, flatMap, and filter / withFilter operations, for-comprehensions look very differently from the implied method calls. As a Scala programmer, you can probably desugar for-comprehensions in your mind’s eye and, as long as your code compiles, all is well.

The problem begins when there’s an error (yes, once in a while this does happen, you know). An error inside a for-comprehension is actually an error inside its desugared form. Highlighting such an error is tricky, because, in addition to the desugaring, this requires some kind of reverse transformation (“error sugaring”?).

In this release, we present a solution that makes errors in for-comprehensions more comprehensible:
for-comprehensions

We treat <- as a reference to the corresponding foreach / map / flatMap method, and we treat if as a reference to the filter / withFilter method.

This also lets you invoke GoTo, Quick Definition, and Quick Documentation actions in those spots. For example:
goto

The binding of <- and if symbols to the method calls makes it possible to display implicit arguments:
implicits

If that looks a bit unusual, it’s due to the limitation of the language syntax – there’s no way to pass explicit arguments to implicit parameters in a for-comprehension (should we create a SIP?). However, this is consistent with the chosen schema, and is required for showing implicits-related errors:
implicit-error

On top of all that, we’ve improved the desugaring of for-comprehensions. So, if you’re curious about what is behind the curtain of syntactic sugar, you can press Alt + Enter and Desugar for comprehension (or you can rely on the Code | Desugar Scala code action).

Find Usages for implicits

Speaking of implicits, previously the implicits hints allowed you to see usages of implicits and go from usages to definitions, but you couldn’t do the reverse – go from definitions to usages, by invoking Find Usages. Well, now you can:find-usages3

This capability relies on bytecode indices, and can also be used to locate other things that are hidden behind the syntactic sugar, and not present in the source code “as is”, including:

  • apply / unapply method calls,
  • Single Abstract Method (SAM) type instantiations,
  • foreach / map / flatMap / filter / withFilter calls via a for-comprehension (this nicely complements the for-comprehenison highlighting).

The feature goes beyond the typical search. While you can emulate Find Usages by means of a full-text search (albeit imperfectly), you cannot rely on text to find something that is not there. Together with the View | Show Implicit Hints, this feature puts you in control of the ‘invisible’ things.

Decompile to Java for Scala .class files

Did you know that IntelliJ IDEA can decompile .class files to Java? Now this is also possible for compiled Scala files. Suppose you have a third-party class that does something, but you don’t know what – because there are no sources at hand:
compiled-class

That’s where the Decompile to Java may come in handy:
decompiled-class

Customizable Scalafmt version

While we supported the scalafmt formatter last year, the Scala plugin could use only a single, bundled version of scalafmt. Now IntelliJ IDEA can automatically download and use whatever version of scalafmt you configure in .scalafmt.conf:
scalafmt

The feature relies on the scalafmt-dynamic module, and so needs an Internet connection. Other than that, this doesn’t require any additional action on your part – everything just works (or at least we hope it does – let us know if it doesn’t for you!).

By the way, we’re also constantly improving the build-in IntelliJ IDEA formatter too :)

Separate HOCON plugin

Hocon is a “Human-Optimized Config Object Notation” format used by many ‘enterprise’ Scala frameworks such as Akka or Play. Because of that, previous versions of the plugin bundled the support of HOCON together with the support of Scala. And yet not everyone who uses Scala needs HOCON.

In this release, we’ve extracted the HOCON support into a separate repository and a separate plugin:
hocon

If you do need HOCON, you don’t have to do anything – the HOCON plugin will be automatically installed with the Scala plugin update (and, subsequently, updated). But now you have more flexibility – if you don’t use HOCON, you can easily disable or remove the HOCON plugin in Settings | Plugins.

(Kudos to Roman Janusz for developing the HOCON plugin!)

Error highlighting improvements

While we’re constantly working to make error highlighting even better, sometimes we improve it a lot… like this time! Here are a few highlights:

  • Partial unification and type variable unification in general.
  • Constructor highlighting, and calls to private constructors.
  • better-monadic-for compiler plugin: implicit0 feature.
  • kind-projector: value level polymorphic lambdas.
  • simulacrum: higher-arity type constructors.

Here’s an example to inspire you:
highlighting

We’re also refactoring our code to make it cleaner and clearer, so if you’ve ever wanted to contribute to the Scala plugin, now is a better time than ever.

While we’ve tried hard to prevent and avoid bugs, absolute perfection is impossible to attain. We’re planning a bugfix release, so please report any issues to YouTrack so that we can fix them as soon as possible.

By the way, the IntelliJ Scala plugin now has an official Twitter account: @IntelliJScala – feel free to follow us for more news and updates.

Develop with Pleasure! No, Drive to Develop! Oh… never mind. Just enjoy!
(and, your feedback is welcome)

Sincerely,

The IntelliJ Scala plugin team

Posted in Uncategorized | 3 Comments

How to use the new features of IntelliJ Scala plugin 2018.2

The recently released IntelliJ Scala plugin 2018.2 offers many new features and improvements. Here’s a tutorial on how to use them:

To play with the implicit-related improvements, paste the following code into Project View:

Posted in New Features | Leave a comment

IntelliJ Scala plugin 2018.2: advanced “Implicit” support, improved patterns autocompletion, semantic highlighting, scalafmt and more

First of all, we want to thank all the contributors who have helped us to implement so many useful new features, bug-fixes, and refactorings. You really inspire us to do our very best work. Your input is greatly appreciated!
Let’s take a look at the new features you’ll find in Scala Plugin 2018.2.

Improved display of Implicit conversions / arguments usage

Now the editor is capable of providing much more useful information about the “implicit” things happening in your code:

  • Scala plugin can show implicit conversions and arguments as inline hints.
  • The Editor also shows hints when an implicit argument is used explicitly.
  • Inline hints provide navigation to the implicit value or function declaration.

IM_1
Note, that sometimes implicit parameters are collapsed to (...). Such folding takes place in case of nested, list of ambiguous arguments, and arguments for conversions.
By the way, ambiguous arguments. We have also re-worked error highlighting. Now, in the same manner, we show places where a function didn’t find an appropriate implicit value, or a set of values is ambiguous.
IM_2
You’ve probably noticed already, that inline hints work in Editor mode. Enable it by
Ctrl + Alt + Shift + “+” hotkey:

  • A second press of the hotkey expands all foldings.
  • Ctrl + Alt + Shift + “-” will collapse all foldings or disable the mode.

Besides the inline hints we’ve also improved:

  • Parameter Info Tooltip (Ctrl/Cmd + P) by adding the implicit parameter;
  • Implicit Arguments Popup (Ctrl/Cmd + Shift + P) now shows type, structure, and location of arguments.

IM_3

Autocompletion: exhaustive pattern matching and pattern completion

The Scala plugin now generates an exhaustive match for sealed types with inheritors, Java Enums, and Scala Enumerations. Together with this, the autocompletion list contains an unapply(...) pattern.
Comp1
Autocompletion in Pattern matching suggests a list of typed patterns for applicable classes and interfaces.
Comp2

Semantic Highlighting support

Now, you can enable semantic highlighting for your project in Preferences/Settings | Editor | Color Scheme | Scala, and assign distinct colors to a function’s parameter, local variable, variable in a pattern-matching case clause, or variable in a sequence comprehension. Then you can scan through a function to track the variable, with no distracting action isolate one of the variables, or switch focus to another variable.
SH

Scalafmt as an alternative to the built-in formatter

The Scalafmt formatter, which used to be its own standalone plugin, is now part of the Scala plugin. It can be configured at Preferences/Settings | Editor | Code Style | Scala. Once the option is set, the reformat action will invoke scalafmt instead of IntelliJ formatter.
Unlike the IntelliJ built-in formatter, scalafmt hasn’t got a lot of Tabs with detailed options, but instead uses a standard way to pass all the settings by a .conf file. If your project contains a .scalafmt.conf file, the IDE will suggest setting scalafmt as the active formatter. But you can also have a .conf file with another name – just specify it in the “Configuration:” setting.
Sometimes scalafmt may be unable to format invalid code. If you want to see warning notifications in the IDE, check this option.
FMT
We have also added the “Reformat on compile” option (enable it in Tab “Other”). The option works for both the IntelliJ built-in and scalafmt formatters.

The release brings lots of other usability enhancements and bug-fixes in Gutter icons, ScalaDoc, and Error highlighting. We would like to say a special thanks to the guys from Wix who improved the Move Refactoring by adding the capability to move object members to another object.
Your feedback is always very welcome. Please do report any bugs you find, and share your ideas, in our issue tracker. Thanks!

Happy developing!

Posted in New Features, Release report | 10 Comments

Ammonite Support

Ammonite, developed by Li Haoyi, is a well-known toolset that lets us use Scala language for scripting purposes. It contains a modernized REPL, a replacement for the Bash system shell, a Filesystem Library, and specific notations for more productive scripting.
Even though many situations in which you would use Ammonite are intended for the command-line mode, quite a few use cases are also relevant for an IDE. After receiving some feedback from our users, we’ve decided to provide advanced support for this technology in our Scala plugin. The set of enhancements includes: support for Ammonite Notations, Run Configuration, a gutter icon for running scripts more easily, and pop-up actions for automatically importing libraries. Read on for more details and screenshots.
0_logo
Continue reading

Posted in New Features | 12 Comments

IntelliJ Scala plugin 2018.1.9: Literal Types, Infix Type Notation, better Error Highlighting

We have exciting news to share with you – the IntelliJ Scala plugin 2018.1.9 is now available! Get the fresh build here or update your plugin via IDEA / Settings / Plugins.

Let’s cut to the chase and take a look at the goodies that are inside this new Release build.

Literal Types support

As literal types are gradually appearing in Scala, we are ensuring their proper handling in our Tools. The Scala plugin now detects when literal types can be used and enables support for them.
LT Continue reading

Posted in New Features, Release report | 1 Comment

IntelliJ Scala plugin 2018.1: inline hints, better Structure View, improved refactoring, and greater usability

Meet the newly released Scala Plugin 2018.1! It’s packed with a whole variety of features, improvements and bug-fixes, and ready for download. In preparing this release for you, we’ve focused on the overall quality and UI/UX aspects, but also managed to add a couple of new interesting features. Read on to find out more.

Parameter Name Hints

If you’ve worked with Java code in IntelliJ IDEA, you’ve probably noticed the feature called “Parameter name hints”. The Scala language has a similar tool called “Named arguments” but we actually cannot completely rely on it. Sometimes developers do not use named arguments in places where they’re useful; and if you call a method from Java, you can not use named arguments at all. That’s why Inline hints are useful for Scala.

You can easily customize when to show such hints in Preferences | Editor | General | Appearance | Show parameter hints | Configure | Scala.

Inline_hints_params Continue reading

Posted in New Features, Release report | 16 Comments

IntelliJ IDEA Scala Plugin 2017.3: Lightbend project starter, Ammonite support, Parallel indexing, and more

First of all, we want to thank all the contributors who helped implement many useful features, bugfixes, and refactorings. You really inspire us to do our very best work. Your input is greatly appreciated!

Now let’s take a look at the new features you’ll find in Scala Plugin 2017.3.

Highlighting of implicit usages

Most likely you are already familiar with this highlighting feature (it uses violet in the Default Theme), which usages of the symbol under the caret across the opened file. Starting with this release, it also highlights places where the target is used implicitly:
Implicit usages highlighting in IntelliJ IDEA

Continue reading

Posted in New Features, Release report | 15 Comments

IntelliJ IDEA Scala plugin 2017.2: SBT 1.0, improved SBT Shell, Play 2.6 and better implicits management

This summer, we are happy to announce a number of new features and improvements in the Scala Plugin. Some of them we decided to introduce in the 2017.1 updates, as soon as they were ready. And the rest are coming to you now – with IntelliJ IDEA 2017.2. We appreciate the contributions of all EAP participants. So, let’s do a brief overview of the recent changes.

Debugging in SBT Shell

Not so long ago we introduced SBT Shell which makes work with SBT project in IntelliJ IDEA more convenient. We continued to improve this feature and can now announce its integration with debugger. With one button click the SBT Shell launches the Debugger server and connects to it.SBT_Debug

Continue reading

Posted in New Features, Release report | 16 Comments

IntelliJ IDEA Scala plugin 2017.1.19: simplified Project View, ScalaTest selection by regexp, improved Akka support

After IntelliJ IDEA 2017.1 was released, we have added many new features to the Scala plugin.

Project View is simplified

We have reworked and simplified navigation to Scala nodes:

  • IDEA 2017-like icon style
  • Type + Companion object nodes
  • Flat package object
  • Files are leaf nodes

To explore the internal structure of a file, enable the “Show Members” parameter in Project View options:

SPhd_1

Continue reading

Posted in New Features, Release report | Tagged , , , | 8 Comments