IntelliJ Scala plugin blog

Statically typed programming languages, such as Java or Scala, are all about types. But not all types are created equal: there’s a difference between how types work in imperative programming and how they work in functional programming. In this article, we show why “imperative” highlighting doesn’t mix well will “functional” code, and present a new, “functional” solution.

The new highlighting is now enabled in the 2019.2 nightly builds of the Scala Plugin, so this update is definitely worth a detailed description. Your feedback is very welcome as always.

TLDR: Before. After. Enjoy!

1. Overexpression of expressions
2. A match for type mismatch
3. The code of code
4. Type ascription to type ascription
5. A diff makes a difference
6. The improvement of improvement

Overexpression of expressions

To illustrate the problem, let’s first consider the following Java code:

Don’t ponder the semantics too much; focus on the syntax. This code contains a type mismatch error: the actual type of the s2 expression (String) doesn’t conform to the expected type – the type of the add method parameter (Integer). As you can see, IntelliJ IDEA highlights such an error with a red squiggly underline. While the particular style depends on the color scheme, it’s of no particular importance – the main point here is that it’s the expression that is highlighted. This way of highlighting type-mismatch errors is the de-facto standard today. And yet it is a legacy of the past, when typical code resembled the snippet above. To IntelliJ IDEA, this mode came from Java and then extended to other supported languages, including Scala.

What’s so special about Java? Typical “good old” imperative Java code is a finely dispersed mix of statements and expressions, which naturally limits the scope of “type mismatch” highlighting. But that’s no longer the case – here’s a Java 8 version of the above code:

In “functional” code, expressions are no longer separated by statements. This doesn’t go well with the legacy highlighting, which is geared toward “imperative” code. Long underlines are hardly pretty, especially if they span across multiple lines. Outer errors mask inner errors. And if you’re in the process of writing such an expression, it would be constantly underlined solely for being incomplete.

All this poses a problem even in Java, let alone in Scala. Given that typical Scala code is more “functional” than Java code, the problem only gets worse. Because a block is an expression in Scala, arbitrary amounts of code, even imperative code, can be underlined “from tip to toe” if we apply the Java recipe directly:

This problem was encountered as soon as we started to implement type-aware highlighting in the Scala plugin. And so, another recipe was borrowed from Java: to highlight the closing brace:

However, this solution is imperfect, for many reasons. First, a single underlined } char is difficult to notice, especially in large blocks of code. Second, the semantics doesn’t match: in Java, this highlighting indicates that a non-void method is missing a return statement, which has nothing to do with the type mismatch (what’s more, you don’t even “return” values from a block in Scala):

The worst of all, though, is that such a solution is partial – it only addresses one special case without addressing the underlying problem. Not every expression is enclosed in a block:

The scheme could possibly be improved by highlighting the final expression instead of the closing brace. This would make the error easier to spot and would get the semantics right. But, again, this only has to do with blocks, and does nothing otherwise. What’s more, this would interfere with writing new code, as each subsequent expression in a block would be underlined.

A match for type mismatch

To find the solution, let’s look closely at the essence of “type mismatch”:

Does the problem lie in the expression? Sometimes you may need to fix the expression to fix the error. Other times, the problem is somewhere else (which is often the case in Scala because of type inference). As the name suggests, “type mismatch” is about a type, and about a mismatch (more precisely, about the actual type and its mismatch against the expected type). Insightful! But why then do we underline the expression? Well, because we need to underline something, and the type is not visible. Whereas the expected type may be present in the code, the actual type is (almost always) inferred, both in Scala and in Java.

But what if we can highlight a type, not an expression? After all, we already use inlay hints to indicate a missing implicit value instead of underlining the expression. We also use the hints to display implicit conversions – a noticeable improvement over the legacy mode. Thus, we can ascribe the actual type, and then highlight it as we see fit:

This way, we can shift the focus from the expression, which is not necessarily the source of the problem, to the actual cause. Even more importantly, that’s how we can limit the scope of highlighting, regardless of the expression length.

The code of code

Just like with the redesign of implicits, it’s crucial that inlay hints represent valid syntax. This lets people employ the same mental model they use to read code. When a notation is uniform and already familiar, the code is the UI, and even complex highlighting looks simple.

But to achieve that, we need cooperation from the language. For example, since Java syntax lacks named arguments, IntelliJ IDEA has to invent the extraneous notation:

Apparently, this cannot be a valid piece of code. Fortunately, Scala supports named arguments out of the box:

The same goes for type ascription. But is the chosen representation indeed a valid piece of code? Let’s ask the scalac:

This code is not valid in the sense that it doesn’t compile, but no wonder – there’s a type mismatch error to begin with. However, instead of complaining about the syntax, the compiler reports exactly the same error in exactly the same place! Thus, the inlay hint behaves as if it were present in the code “as is”.

Apart from the syntax, there’s the question of style. Originally, parameter name hints look different from code. This is somehow justified in Java, where hints don’t represent code. In Scala, however, hints represent valid code. What’s more, Scala hints behave like code: they provide error highlighting, code folding, tooltips, navigation, and brace matching (and we use the current code style for all of that). When we designed the implicit hints, we unanimously voted to display them “like code”, because this feels more natural. It makes sense to do the same with the type mismatch hints.

It also makes sense to display the existing type annotation hints in the same way, for uniformity. This is also more in line with the implicit hints. As a nice bonus, all the hints then support navigation, folding, tooltips, and brace matching:

The monospaced font is slightly wider, but this is compensated by the added dynamic folding and by showing type hints only when truly needed.

Interestingly, type mismatch hints are not just for type mismatch – they also provide editing assistance. Type ascription for expressions works in the same way as the type annotations for definitions (which often are just named expressions). Both kinds of hints complement the on-demand Type Info action.

Type ascription to type ascription

Showing type ascription as though it was present in the code is great and all, but what if it actually is present? Previously, we highlighted a typed expression as any other expression. That is, we underlined it:

But what are we supposed to do now? Should we ascribe a type twice? While Scala lets you do that – (expression: Actual): Actual, there’s a better way. If you think about it, a type ascription hint is just a blank canvas for error highlighting. If the canvas is already present, so much the better – we can do the highlighting right away:

When some expression lacks an ascribed type, the Scala compiler first has to infer it, and then check for a possible type mismatch. But when a type is already present, the compiler can skip the type inference, and get right to the type checking. That’s what we do now, and this perfectly matches the highlighting of inlay hints (interestingly, this also reduces the scope of type mismatch underlines even without using inlay hints).

So far so good, but what if an ascribed type is incompatible? For example:

Previously, we highlighted the expression as though the type ascription was in fact a type annotation. Should we now ascribe an actual type as (expression: Actual): Incompatible? Not really. There’s a difference between type annotation and type ascription. The former is added to declarations and definitions, and corresponds to explicit types in Java. The latter is added to the expression and corresponds to the cast operator in Java (but that supports only upcasts). In the case of type annotation, the type goes first and the expression follows. By contrast, with type ascription, the expression goes first and the type follows. So, type ascription always specifies the actual type; it’s just that the type might be incompatible. When an expression cannot be upcast to a given type, it’s the type that is the culprit:

That’s how we can reduce the scope of highlighting and keep the highlighting uniform (again, this would work even without using inlay hints).

A diff makes a difference

We’ve done a good job so far. But we need to go deeper narrower. Many Scala types comprise other types and those components are often the original source of a type mismatch. Previously, we compared types as indivisible units:

While technically this wasn’t wrong, it was hardly convenient, because you had to “parse” and “type check” the constituent types in your head to understand why exactly the types do not match. Now, we highlight only the truly “mismatching” parts of a type:

We can even show a fine-grained, vertically-aligned tooltip for a pairwise comparison:

It’s not just a text-based diff either – we take syntactic sugar, subtyping, variance, and other stuff into the account.

The improvement of improvement

That’s the big picture. There are also some special cases, for example, the handling of literal types or method invocations. Although the main work has been completed, the polishing is still in progress. Please use nightly builds for the the most recent updates, and report any bugs or suggestions to YouTrack.

On the whole, it seems that the new highlighting is a substantial improvement over the legacy, “imperative” scheme, and the more “functional” your code, the bigger the improvement. We hope this feature will prove useful for Scala, and we expect other IntelliJ IDEA languages to follow

For several years we’ve been working with Triplequote to help them develop the Hydra IntelliJ IDEA plugin. By combining IntelliJ IDEA with the Hydra parallel compiler, you can speed up both Scala development and Scala compilation. Here is a tutorial from the Triplequote team to explain the details.

1. What is Hydra?
2. Install the Hydra IntelliJ IDEA plugin
3. Compile!
4. Low memory detection
5. Compilation bottlenecks detection
6. What next?

What is Hydra?

Hydra, developed by Triplequote, is the only Scala compiler that parallelizes compilation across all available compute cores, speeding up compilation time up to 5x, and providing much faster return-to-productivity.

In addition to faster return to productivity, Hydra ships with built-in compile time monitoring, allowing you to crack open the compiler “black box” with the Hydra Dashboard, providing visualizations and metrics for optimizing your project’s compilation speed over time, and helping teams keep performance under control.

Hydra supports the full language (macros and compiler plugins – Scala.JS included), and it smoothly integrates with your build tool of choice, whether that is sbt, Maven or Gradle.

This article shows how to set up Hydra in IntelliJ IDEA and the many benefits delivered by Hydra in addition to parallel compilation.

Install the Hydra IntelliJ IDEA plugin

To install the Hydra IntelliJ IDEA plugin just search for “Triplequote Hydra” on the Plugins Marketplace.

Once installed, open the “Hydra Compiler” page and notice that at the top it says “No license detected”.

This is normal, as you need a license to compile with Hydra. Go ahead and hit the button “Get trial license” located on the top-right ([1]), and provide your full name and email. You will immediately receive the license key (check your SPAM folder if you can’t see it).

To use it, go back to the “Hydra Compiler” page, and click the “Enter license key” button ([2]). Paste the Hydra Developer license key you have received and proceed. The “Hydra Compiler” page is updated to confirm the validity of the license.

Congratulations! You are all set and ready to compile your Scala projects with Hydra.

Compile!

All Scala projects opened in IntelliJ IDEA will now be compiled with Hydra. You don’t need to do anything, it just works!

To try it out, hit “Build > Rebuild project” and notice in the Message view that “N Hydra workers” were used to compile your Scala sources.

Keep in mind that the default number of Hydra workers used may vary depending on the machine’s hardware, what type of Hydra license is in use, and the number of compiled sources. You can always modify it yourself on the settings page.

Low memory detection

A common (and often ignored) reason for long compilation time is that not enough memory heap is reserved for compilation. In fact, Scala compilation is allocation hungry and can put considerable pressure on the JVM Garbage Collector (GC), to the extreme that more time may be spent GC-ing rather than compiling!

The default heap size reserved for Scala compilation inside IntelliJ IDEA is 1024MB (see the “Scala Compiler Server” preference page under “Preferences” > “Build, Execution, Deployment” > “Compiler” > “Scala Compiler” > “Scala Compile Server”). While this default is good to get started, as modules grows in size and complexity more memory is often needed. Hence, memory usage needs to be constantly monitored during compilation, as otherwise it’s easy to miss that a reason why compilation takes long is because a sizeable amount of time is wasted GC-ing.

The solution to this rather annoying problem is to monitor GC. This can be achieved with the help of a profiler (e.g., VisualVM), but we prefer to automate and when using Hydra to compile there is no need to profile GC, ever. In the event that a considerable amount of time is spent GC-ing during compilation, Hydra reports a warning so that you can act on it.

To fix the problem, increase the max heap size in the “Scala Compiler Server” preference page. Be aware that for the change to be picked up the Compiler Server process needs to be restarted – click the throttle icon located at the bottom of the editor, then “Stop” and “Run” again.

Compilation bottlenecks detection

We, as Scala developers, have a tendency to accept our project’s compilation time without ever questioning it. After all, what could we possibly do about it? It turns out there is actually a lot we can do, but to direct our efforts we need to gain visibility into what the compiler does and know where the biggest bangs for the bucks are. Said otherwise, we need to know what Scala sources in our project takes the longest to compile.

One of the greatest features of Hydra is its built-in compile-time monitoring capabilities, and once again Hydra does the heavy lifting for you: if it detects a source file takes more than 5 seconds to compile, it reports the information at the end of the compilation.

It’s surprising how having this simple piece of information changes entirely the attitude we have in the face of the compiler. All of a sudden, we are surprised that some sources may be taking 5, or 10 seconds, or even more to compile, and hence we dig deeper. How do we do that? And how to know what source(s) takes long to compile? We look at the compilation data collected in the Hydra Dashboard!

The compilation metrics above were collected while compiling the Doobie open-source project. The Dashboard points out immediately what source took more than 5 seconds to compile.

This source file contains 250+ case classes definitions. For each case class, the Scala compiler generates several methods in both the class and its companion object. So, while a single case class is very fast to compile, the time adds up when you have over 250+ defined in a single source file. But using case classes was a design decision for the library, so in this case we are ok with the implied compile time cost. This is a remarkable insight we just gained on the project!

Despite the above considerations, there is still value in breaking up a source file that takes a long time to compile, as this reduces the incremental compilation time when editing the specific source. Thanks to IntelliJ IDEA refactoring capabilities, this is easy to achieve, as we can use the “Move” refactoring to split the logic into multiple files, bringing the compilation time of all sources below 5 seconds (the smaller the better!).

As you start exploring the compilation metrics collected on your project, you’ll discover bottlenecks you didn’t expect. A common one is to have imports in scope that are used to generate implicit values through macros (for instance, using the Circe automatic derivation feature). To learn more on this specific topic and how to address it, you may want to watch this ScalaDays talk “5 Things you need to know about Scala compilation” or read the Zalando article “Achieving 3.2x faster compilation time”. The Hydra Dashboard documentation also provides additional useful advice for speeding up compilation further.

What next?

Check out the Hydra documentation to find out more about Hydra and discover how easy it is to integrate it with your build tool of choice. Also, if you would like to benchmark how Hydra performs against the vanilla Scala compiler, you might find the hydraBenchmark command very useful!

And don’t forget to install the Dashboard to quickly identify compilation bottlenecks affecting your Scala projects!

Some releases are about adding more features. In this release, we’ve focused on improving the existing features of the Scala plugin. These features are the bread and butter of Scala programmers, so we hope the improvements will make our day-to-day work much easier. Here goes:

1. Highlighting of for-comprehensions
2. Find Usages for implicits
3. Decompile to Java for Scala .class files
4. Customizable Scalafmt version
5. Separate HOCON plugin
6. Error highlighting improvements

Highlighting of for-comprehensions

For-comprehensions can sometimes be… incomprehensible. Being a syntactic sugar for composing foreach, map, flatMap, and filter / withFilter operations, for-comprehensions look very differently from the implied method calls. As a Scala programmer, you can probably desugar for-comprehensions in your mind’s eye and, as long as your code compiles, all is well.

The problem begins when there’s an error (yes, once in a while this does happen, you know). An error inside a for-comprehension is actually an error inside its desugared form. Highlighting such an error is tricky, because, in addition to the desugaring, this requires some kind of reverse transformation (“error sugaring”?).

In this release, we present a solution that makes errors in for-comprehensions more comprehensible:

We treat <- as a reference to the corresponding foreach / map / flatMap method, and we treat if as a reference to the filter / withFilter method.

This also lets you invoke GoTo, Quick Definition, and Quick Documentation actions in those spots. For example:

The binding of <- and if symbols to the method calls makes it possible to display implicit arguments:

If that looks a bit unusual, it’s due to the limitation of the language syntax – there’s no way to pass explicit arguments to implicit parameters in a for-comprehension (should we create a SIP?). However, this is consistent with the chosen schema, and is required for showing implicits-related errors:

On top of all that, we’ve improved the desugaring of for-comprehensions. So, if you’re curious about what is behind the curtain of syntactic sugar, you can press Alt + Enter and Desugar for comprehension (or you can rely on the Code | Desugar Scala code action).

Find Usages for implicits

Speaking of implicits, previously the implicits hints allowed you to see usages of implicits and go from usages to definitions, but you couldn’t do the reverse – go from definitions to usages, by invoking Find Usages. Well, now you can:

This capability relies on bytecode indices, and can also be used to locate other things that are hidden behind the syntactic sugar, and not present in the source code “as is”, including:

• apply / unapply method calls,
• Single Abstract Method (SAM) type instantiations,
• foreach / map / flatMap / filter / withFilter calls via a for-comprehension (this nicely complements the for-comprehenison highlighting).

The feature goes beyond the typical search. While you can emulate Find Usages by means of a full-text search (albeit imperfectly), you cannot rely on text to find something that is not there. Together with the View | Show Implicit Hints, this feature puts you in control of the ‘invisible’ things.

Decompile to Java for Scala .class files

Did you know that IntelliJ IDEA can decompile .class files to Java? Now this is also possible for compiled Scala files. Suppose you have a third-party class that does something, but you don’t know what – because there are no sources at hand:

That’s where the Decompile to Java may come in handy:

Customizable Scalafmt version

While we supported the scalafmt formatter last year, the Scala plugin could use only a single, bundled version of scalafmt. Now IntelliJ IDEA can automatically download and use whatever version of scalafmt you configure in .scalafmt.conf:

The feature relies on the scalafmt-dynamic module, and so needs an Internet connection. Other than that, this doesn’t require any additional action on your part – everything just works (or at least we hope it does – let us know if it doesn’t for you!).

By the way, we’re also constantly improving the build-in IntelliJ IDEA formatter too

Separate HOCON plugin

Hocon is a “Human-Optimized Config Object Notation” format used by many ‘enterprise’ Scala frameworks such as Akka or Play. Because of that, previous versions of the plugin bundled the support of HOCON together with the support of Scala. And yet not everyone who uses Scala needs HOCON.

In this release, we’ve extracted the HOCON support into a separate repository and a separate plugin:

If you do need HOCON, you don’t have to do anything – the HOCON plugin will be automatically installed with the Scala plugin update (and, subsequently, updated). But now you have more flexibility – if you don’t use HOCON, you can easily disable or remove the HOCON plugin in Settings | Plugins.

(Kudos to Roman Janusz for developing the HOCON plugin!)

Error highlighting improvements

While we’re constantly working to make error highlighting even better, sometimes we improve it a lot… like this time! Here are a few highlights:

• Partial unification and type variable unification in general.
• Constructor highlighting, and calls to private constructors.
• better-monadic-for compiler plugin: implicit0 feature.
• kind-projector: value level polymorphic lambdas.
• simulacrum: higher-arity type constructors.

Here’s an example to inspire you:

We’re also refactoring our code to make it cleaner and clearer, so if you’ve ever wanted to contribute to the Scala plugin, now is a better time than ever.

While we’ve tried hard to prevent and avoid bugs, absolute perfection is impossible to attain. We’re planning a bugfix release, so please report any issues to YouTrack so that we can fix them as soon as possible.

By the way, the IntelliJ Scala plugin now has an official Twitter account: @IntelliJScala – feel free to follow us for more news and updates.

Develop with Pleasure! No, Drive to Develop! Oh… never mind. Just enjoy!
(and, your feedback is welcome)

Sincerely,

The IntelliJ Scala plugin team