Big Data Tools 0.0.356: Minor Improvements and Stability Fixes

Last week we announced the very first version of our new IntelliJ IDEA Ultimate plugin aimed at providing tools for working with Big Data. Currently, the plugin’s functionality is limited to the support for Scala and built-in integration of Zeppelin notebooks. A lot more functionality is to come, including the support for Spark, Hadoop, S3, etc.

Today we’re glad to release a little update (v0.0.356) that brings minor but important bug-fixes and improvements. Below is a list of the changes included:

  • Overall execution progress for Zeppelin notebooks (works only for 0.8.x and higher)

  • The Structure View popup for Zeppelin notebooks

  • In the bottom of the cell, it now displays the author, execution time, and modification time
  • The Basic HTTP authentication issue in Zeppelin connection dialog (it overwrote the Login field, BDIDE-402)
  • The experimental support for Zeppelin 0.9.0 (BDIDE-392)
  • Connection is re-established automatically after a restart of an interpreter
  • The default height of charts is now 10 rows
  • The text output inlays are now not scrolled to the end (as it used to)
  • The issue with processing the results that contain Tab characters (BDIDE-383)

That’s it for now. As we promised, more goodies are on the way. Stay tuned, and in case you haven’t tried the plugin yet, make sure you do. Any feedback is very welcome here in the comments or in our tracker.

P.S. Please join our warm community in Slack to get instant support, tips, updates, and talk to other users of the plugin, Zeppelin, and Spark!

Posted in New Features | Tagged , , | Leave a comment

Meet Big Data Tools – Spark Integration and Zeppelin Notebooks in IntelliJ IDEA

Hooray! Today we have some exciting news for you. After all, it is not often JetBrains introduces new developer tools. Now that you’re excited as well, we’re very pleased to announce Big Data Tools – a new IntelliJ IDEA plugin that integrates Spark and brings support for editing and running Zeppelin notebooks. Now with the Big Data Tools plugin, you can create, edit, and run Zeppelin notebooks without ever having to leave your favorite IDE. The plugin offers smart navigation, code completion, inspections & quick-fixes, and refactoring inside notebooks.

So, what precisely does the plugin support now?

  • Browsing, creating, and deleting notebooks
  • Coding assistance for Scala paragraphs
    • Code completion
    • Rename refactoring
    • Extract variable, method or parameter
    • Go to declaration
    • Show usages
  • Creating and deleting paragraphs
  • Running paragraphs
  • Browsing paragraphs’ output
  • Support for basic visualization

Note coding assistance is currently limited to the Scala code. Other languages will be coming later. Also, in the future, we plan to go beyond Zeppelin notebooks and add more general features that will streamline the developer experience for data engineers, data scientists, and other professionals that work with Big Data.

If you prefer to see it in action rather than just read about it, make sure to watch this short video:

Otherwise, go ahead and try it out for yourself! Below is a short introduction into how to get started. Continue reading

Posted in EAP, New Features | Tagged , , | 8 Comments

Integrating Developer Experiences — The Build Server Protocol in the IntelliJ Scala plugin

IntelliJ IDEA is an IDE – an Integrated Development Environment. I want to focus on the “Integrated” in this post, which is based on my Scala Days talk.

Many developers believe in the “one tool, one job” UNIX philosophy of tooling: A small set of orthogonal, powerful command line tools that individually only do one job, but do it well, and connect through a common interface: stdin/stdout streams. This is great for composing scripts and automating common tasks. But taking a closer look at some of these tools, this breaks down. How often do you have to look up the command line options of find? Myself, it’s almost every time I use it. And each tool has its own idiosyncratic options.

The IDE follows a different approach: one tool, all the jobs. To offer a unified, consistent interface for all the tasks related to software development. Naturally, this leads to big monolithic applications that are hard to embed in an automated pipeline. This often means we need to duplicate some of the configuration around building and running our software in different environments: from the desktop, in a CI pipeline, and finally, in production. But for most of us, this is worth it for the effective developer experience.

Integration rationale

Traditionally IDEs, including IntelliJ IDEA, come with their own versions of a compiler, project configuration, artifact assembly, and so on. That works fine for projects developed by a single programmer, or a small team, but it doesn’t play well with an automated continuous integration and delivery pipeline. We could manually recreate the configuration for the desktop environment, but we both know that it will take about 3 commits for them to get out of sync and cause you a lot of headache trying to figure out where that ClassDefNotFoundError is coming from that is only reproducible on your machine.

The obvious remedy is to write integrations for popular tools and frameworks to import their configuration and map it to the built-in function, taking the responsibility of maintaining the local version off your hands. That works, right? At least, until the next tool or framework you’re eager to try comes along. Which means another IntelliJ IDEA integration. And lots of bug fixing until it works smoothly.

Integrating build tools

Each tool integration is purpose-built on whatever interfaces the tool in question offers. Let’s have a look at the IntelliJ IDEA sbt support. On the frontend side, we have:

  • A menu option to import sbt projects
  • The sbt project toolwindow
  • The sbt preferences pane
  • The sbt shell toolwindow

Not too complicated on the surface. But to make this work, we have introduced a load of background complexity:

  • A special sbt plugin that extracts the project structure and dumps it into an xml file …
  • … that is then parsed by the Scala plugin and transformed into the IntelliJ IDEA project model.
  • Another sbt plugin that modifies the sbt shell output so that IntelliJ IDEA understands when a task is running or has been completed.
  • A hacky mechanism to inject the plugin into your build, so that you don’t have to go through additional configuration steps.
  • Lots of glue code to facilitate the communication between thr IntelliJ IDEA and sbt processes.

Needless to say, this is brittle and breaks left and right. It’s worth it for sbt because of the large user base. But Scala users never seem entirely content with their build tools; someone decides to write a new every year or so. We would need to repeat all the effort we put into supporting sbt for every one of them to provide a comparable experience. How can I justify this for every new tool that, on its own, is only used by a small fraction of our users? On the other hand, the build tools also suffer from a chicken-and-egg problem. Users have come to expect their tools to work together. How are new tools going to get any traction without decent editor or IDE support?

Integral approach

Prior Art

Luckily, there is some prior art on solving this problem. Microsoft’s Visual Studio Code editor popularized the Language Server Protocol (LSP), which addresses the problem of programming language compiler frontends (language servers) talking to editors to provide rich editing features that otherwise would need dedicated editor-side support for a language. While previously, every editor or IDE would need to implement this as language-specific compiler integrations, LSP allows most of the functionality to be implemented in a standardized way. The idea itself is not new, LSP was inspired by other language server approaches such as OmniSharp and the TypeScript Server.

Language servers are also not new to the Scala ecosystem: ENSIME used to be a popular alternative to IDEs.

But LSP has been uniquely successful in terms of adoption. So much so, that people keep asking us if we are going to change the IntelliJ Scala plugin to use it rather than our own implementation of a Scala parser and typechecker, in an effort to address many long-standing highlighting problems. The short answer is probably not, due to various technical issues. There may be space for a hybrid approach, but that is a topic for another post.

Build Server Protocol

At the Scala Center, Jorge Vicente Cantero and Ólafur Páll Geirsson were having a similar problem integrating the various Scala build tools with an editor frontend. Bloop is a build server that aims to optimize compilation times for Scala. It offers a standard implementation of not only the incremental compiler, but also the plumbing of keeping a hot JVM available for compiles and various other optimizations that depend on the structure of the code. At first, bloop was a command line tool that you had to invoke separately from your build tool or editor of choice. As discussed above, this is not ideal from a developer ergonomics point of view.

Naturally, when Jorge proposed the idea of a “Build Server Protocol”, I was very interested. One protocol to build them all? That would solve so many of my problems. So we started collaborating on developing and implementing the protocol. Now, a year after the original announcement, with BSP v2.0 being basically stable and most of the kinks in the implementations in Bloop, Metals and IntelliJ IDEA ironed out, we feel confident to announce it as an officially supported feature in the 2019.2 release of the IntelliJ IDEA Scala plugin.


The primary implementations of BSP at this moment are the Scala plugin for IntelliJ IDEA and Metals as clients, and Bloop as server. Server support in other tools is in the works.

IntelliJ Scala as client

The BSP implementation in IntelliJ IDEA allows importing projects from BSP servers and running compilation over the protocol. Unlike previous build tool integrations, this requires no knowledge of the build tool specifics from the IntelliJ IDEA side, nor does it need any IntelliJ-specific support by build tools implementing the protocol.

Since compilation is handled by the build tool, there is no mismatch between the compiler output of IntelliJ IDEA and that of the build tool, even for projects where the project model cannot be successfully imported into the IDE. Note, however, the error highlighting within the editor is still done by the Scala plugin itself, so in some cases there may be red squiggles where there shouldn’t be. We have some ideas on how to improve this by reusing build server messages to inform the highlighting. This would have implications on the features we can offer in the highlighting, so it’s not completely straight-forward.

To use IntelliJ IDEA with Bloop, follow our instructions.

Bloop as server

Bloop is not a full-fledged build tool in itself, but rather a compile server that focuses on compiling, testing and running Scala code. There are plugins for build tools, such as sbt, Mill, Maven, or Gradle, that export the respective project definitions to Bloop config files. This has the benefit of being an easy way to add BSP support to existing tools, even if they don’t have a built-in server. The drawback of this is that importing into the IDE requires a two-step tools-specific process on the user’s side: First, exporting the structure, second, importing it.

But the server nature of Bloop allows another novel use: As a compilation backend to an existing build tool. Instead of writing a tool-specific compiler, usually based on the Zinc incremental compiler library, build tool authors can simply leverage Bloop via BSP. Fury takes this approach.

Further work

Fury is a build tool and dependency manager currently under development by Jon Pretty and VirtusLab. It completely outsources the job of compiling Scala to Bloop, acting as a BSP client. At the same time, it already offers a simple BSP server implementation that allows importing a Fury build into IntelliJ IDEA. With further development, it will support compilation over the protocol by passing through compilation events from Bloop.

Mill is a relatively new build tool by Li Haoyi. It has its own mechanism for exporting builds to IntelliJ IDEA, but this uses an outdated format. It also comes with a module that allows exporting to Bloop, which enables an IntelliJ IDEA import via BSP, at the cost of a two-step process. Neither of these options allow taking advantage of the built-in Mill task graph, which is why we are now working on Mill-native BSP support.

Pants is a build tool by Twitter geared towards large “monorepo” style codebases and designed to build multiple languages. Twitter is currently working on BSP support for Pants.

The Build server protocol is already being used for in-house build tools, allowing a level of IDE integration that would otherwise be hard to achieve and require dedicated plugins.

sbt does not yet incorporate a BSP implementation, but it is possible to build sbt projects over BSP by exporting them to Bloop. In the future, we aim to improve the IntelliJ IDEA sbt support by running it over BSP, which will enable a more robust communication mechanism that the current integrated sbt shell.

Library: bsp4j

Most bsp implementations are based on the bsp4j library, a simple Java library based on lsp4j. The repository comes with a testkit and some tests that help to understand the usage of the library.


The design of the Build Server Protocol closely follows the Language Server Protocol, and reuses some of its data types, such as Diagnostic. This makes it easy to use BSP as a back-end to a language server, or even implement the two protocols side-by-side in a single server.

Like LSP, BSP is a JSON-RPC protocol — a client-server protocol that nonetheless allows bi-directional communication between client and server. Data may be exchanged both by requests and notifications. Requests are typically sent by the client to request some data about the build, or ask the server to execute an action, such as compile or test. Notifications are more often sent by the server to give updates about the current state of a task, or report compilation problems (diagnostics).


I’m going to give a brief overview of some of the main BSP concepts to give you an idea of how it works and an orientation for implementation, as well as considerations for the design. If you are interested implementing BSP, please refer to the full BSP specification.

Server discovery


Before talking to each other, the IDE and build server need to know how to find each other. In LSP, this is usually handled by a language-specific client implementation that knows how to start the right language server. This was not quite good enough for our purposes. We wanted any client to be able to connect to any server and offer basic functionality out of the box, without requiring a dedicated plugin.

Our solution is the BSP connection protocol. It’s quite simple:

  1. The server places a JSON connection file

    into the {workspace}/.bsp/ or a OS-dependent system or user directory:

  2. The client discovers this file, and starts the server process.
  3. Client and server write and read JSON-RPC messages over stdin/stdout of the server process.

This very simple cross-platform mechanism enables a zero-configuration approach: no pipes, ports, or IPs necessary. Of course the server process is free to implement a more advanced protocol behind the scenes, including remote communication. But the client doesn’t need to know about it.

Build structure


A major part of the Build Server Protocol is to standardize the representation of the build graph. Most build tools have some way of modeling nodes in a build graph in term of components: sbt calls them projects, IntelliJ names them modules, in Bazel they’re targets. Depending on the tool, these nodes may be a different level of granularity, and don’t always map to another tool’s idea of a node. Nonetheless, they all share some commonalities that BSP unifies under the concept of targets, closely following the Bazel design:

  • Targets may depend on one another, forming a directed acyclic graph.
  • A target contains some metadata about the language and language level it is compiled with, and possible actions on this target (such as compile or test).
  • A target can be associated with any number of source files or directories
  • Sources associated within a target are compiled as a unit.

The build targets in a workspace are requested in a single workspace/buildTargets request from the client to the server, while the associated sources are requested separately in the buildTarget/sources request.

In BSP, a set of sources may be associated with any number of targets. This is in contrast to the IntelliJ IDEA project model, where a source file may only be part of a single module. This limitation exists to allow the IDE to always know how a file needs to be highlighted, and is tightly coupled to the implementation of many of IntelliJ IDEA’s internals. This has always been a challenge when importing sbt projects that cross-compile sources to several targets (typically scala.js or Scala Native), as there is no straightforward mapping of shared sources to the IntelliJ IDEA model. Our workaround is to create synthetic “shared” modules that are in turn dependencies of the actual modules sharing the sources. This works at least some of the time, but will often have unexpected results. We are looking at better ways to support shared sources in the future.



The other major benefit of a build protocol is executing typical build server actions in a standard way. The most common one of course is compilation. Without any tool integration, this is where you’d switch to the terminal to type a command whenever you want to know about all the programming errors you’ve made. But what you’d rather do, is to just press a button in the IDE, or let it happen every time you save a file. To this end, BSP offers some requests.

The buildTarget/compile request is sent from the client to the server with parameters for which targets the server should perform a compilation. Once completed, the server will reply with a CompileResult that contains a status code about the compilation result. On its own, this isn’t very informative, so during the compilation, the server may send any number of diagnostics and progress notifications (see below)

Analogously, buildTarget/test requests running tests within some modules, and will return with a TestResult. While the request is being run, any number of test-specific progress notifications may be sent from the client to the server.



As in LSP, diagnostics give information about issues at certain code points — compiler warnings and errors. Diagnostics may be sent at any time from the server to the client as build/publishDiagnostics notification, but usually they will be associated with a compile request.

A point to note: One diagnostics notification always contains all the individual diagnostics that are currently valid for a single file. Subsequent notifications for the same file invalidate the previous ones. This allows the editor to persistently highlight all the currently valid errors, and clear them on the next notification if they are fixed.


Often tasks in a build server will be long-running and involve many sub-tasks. We added task progress notifications to the protocol to let the client give informative feedback on the state of a build.


Task notifications are sent from the server to the client to give updates on the running tasks. Tasks notifications can be one of three types:

  • build/taskStart signals the beginning of some task, such as compilation of a target or running of a test.
  • build/taskProgress gives updates on how far the task has progressed.
  • build/taskFinish signals the completion of a task.

When a server implements these notifications, running a build will look like this in IntelliJ IDEA:


Towards an Integrated Developer Experience

The Build Server Protocol implementation in the Scala plugin for IntelliJ IDEA helps you to be productive right now with Bloop, or to get hacking on Fury. So far the implementation is still limited to Scala targets. There’s still plenty of work to be done to create a fully Integrated Developer Experience.

Looking at only Scala projects, some of the implementation is still incomplete, but it is improving. We are currently working on running tests over BSP In IntelliJ IDEA, which will make it easier for test tool authors to offer IDE support.

Building on the base protocol, it will be easier to offer robust language- and tool-specific support for sbt, Mill, and others in the future. This could include editor support for build files, or running arbitrary tasks.

Once the protocol is established in the Scala ecosystem, we want to take a closer look at how it can help with polyglot tools like Bazel and Pants as well as within other language ecosystems.

Get involved!

We’d love for you to get involved and help us to improve the experience you have with build tool and editor support! The simplest way is to try it for yourself and give us feedback.

If you want to get your hands dirty, work directly on the client implementation in IntelliJ Scala or contribute to existing servers, or create new implementations in Bloop, Mill, Fury, sbt, or others and chat on the BSP Gitter channel


Posted in New Features | Tagged , | 1 Comment

IntelliJ Scala Plugin 2019.2: Functional Code Highlighting, Fine-grained Type Diff, Case Clause Completion, Build Server Protocol, and More

The 2019.2 release of the Scala plugin offers many improvements in different areas. The key focus of this release is a new code highlighting paradigm whereby the highlighting is now cleaner, yet provides even more coding assistance.

1. Functional code highlighting
2. Fine-grained type diff
3. Interactive type hints
4. Wrap / unwrap expressions as you type
5. Unobtrusive redundant code inspections
6. Case clause completion
7. Build Server Protocol
8. Language injection for multiline strings
9. Duplicate code fragment inspection

Functional code highlighting

Previously, IntelliJ IDEA highlighted type mismatch errors with a red squiggly underline. But long underlines are hardly pretty, especially if they span across multiple lines. Outer errors mask inner errors. And if you’re in the process of writing such an expression, it would be constantly underlined solely for being incomplete.

Now we can do better: instead of underlining the whole expression, the Scala plugin now shows a type ascription hint and highlights only the part that doesn’t match the expected type:

This significantly cleans up your editor, while providing even more coding assistance. To learn about the “whys” and the “hows”, see our recent post, Functional Highlighting for Functional Programming. (Note that the mode is controlled by Settings | Languages | Scala | Type mismatch highlighting.)

Fine-grained type diff

Previously, we compared types as indivisible units. While technically this wasn’t wrong, it was hardly convenient, because you had to “parse” and “type-check” the constituent types in your head to understand why exactly the types did not match.

Now we can show a fine-grained, vertically-aligned tooltip for a pairwise comparison:

It’s not just a text-based diff – the algorithm takes syntactic sugar, subtyping, variance, and other tricky stuff into the account.

Interactive type hints

Type annotation hints now support dynamic folding, tooltips, navigation, and brace matching:

What’s more, type hints are shown only when they are truly needed (so, if you previously disabled type hints because they were too noisy, consider giving them another try). That’s the second editor cleanup, alongside with the new type mismatch highlighting.

Wrap / unwrap expressions as you type

Before, you had to invoke the Add braces around single line intention to wrap an expression in curly braces, or the Remove braces intention to unwrap an expression.

Now, IntelliJ IDEA can do that automatically, as you type:

While the intentions are still available (and work just fine), the automatic balancing is much more discoverable and is very handy in practice.

Unobtrusive redundant code inspections

IntelliJ IDEA has always highlighted unused imports or unreachable code as “unused” (as defined by the Color Scheme). Other things, however, were highlighted as “generic warning or error”, which polluted code and concealed actual errors.

Now we highlight all redundant code uniformly, so that the highlighting looks unobtrusive:

That’s the third editor cleanup. Since the highlighting now doesn’t get in your way, we can implement many more useful related inspections. Feel free to vote for the ones you like the most!

Case clause completion

The Scala plugin has already offered exhaustive match completion for quite some time.

Now you can also complete individual case clauses:

What’s more, the completion now works in partial functions, as well as in match expressions.

Build Server Protocol

The Build Server Protocol (BSP) defines a standard way for build tools to talk to IDEs, allowing you to open projects, run compilation, perform tasks, display error messages, and monitor progress updates directly in the IDE.

For a while, BSP support was an experimental feature, but not anymore – now it’s fully supported:

The supported build tools include Bloop and Fury so far. Support for Mill is coming soon. We’re going to post more details on the BSP integration soon, so stay tuned.

Language injection into multiline strings

Now let’s talk about a few features that are only available in IntelliJ IDEA Ultimate. The first one is the support of language injection into multiline strings:

This feature may come handy for embedding SQL, HTML, or JavaScript into Scala code.

Duplicated code fragment inspection

Did you read the HTML comment in the previous release announcement? If you didn’t, here it is again: “We’ve also implemented duplicates detection, but this capability is not yet enabled in the IntelliJ Platform; expect that feature in the next release”. So, here goes:

Because the feature is implemented as an inspection, you can invoke Run inspection by name | Whole project to fix all the duplicates at once. The algorithm can filter out whitespaces, syntactic sugar, and other accidental details. So, don’t repeat yourself, don’t repeat yourself.


The new features work in synergy with the implicit hints, which have already cleaned the editor from pervasive “implicit underlines”. The Scala plugin now offers a unique combination of clarity and power: your code looks almost as clean and unclouded as in a plain text editor, while you have intelligence that even the Scala compiler can’t provide, including implicit hints, fine-grained type diffs, inspections, and much more.

Your feedback is very welcome as always. Please report any bugs to YouTrack so that we can fix them as soon as possible. Thank you!


The IntelliJ Scala plugin team

Posted in New Features, Release report | 8 Comments

Functional Highlighting for Functional Programming

Statically typed programming languages, such as Java or Scala, are all about types. But not all types are created equal: there’s a difference between how types work in imperative programming and how they work in functional programming. In this article, we show why “imperative” highlighting doesn’t mix well will “functional” code, and present a new, “functional” solution.

The new highlighting is now enabled in the 2019.2 nightly builds of the Scala Plugin, so this update is definitely worth a detailed description. Your feedback is very welcome as always.

TLDR: Before. After. Enjoy!

1. Overexpression of expressions
2. A match for type mismatch
3. The code of code
4. Type ascription to type ascription
5. A diff makes a difference
6. The improvement of improvement

Overexpression of expressions

To illustrate the problem, let’s first consider the following Java code:


Don’t ponder the semantics too much; focus on the syntax. This code contains a type mismatch error: the actual type of the s2 expression (String) doesn’t conform to the expected type – the type of the add method parameter (Integer). As you can see, IntelliJ IDEA highlights such an error with a red squiggly underline. While the particular style depends on the color scheme, it’s of no particular importance – the main point here is that it’s the expression that is highlighted. This way of highlighting type-mismatch errors is the de-facto standard today. And yet it is a legacy of the past, when typical code resembled the snippet above. To IntelliJ IDEA, this mode came from Java and then extended to other supported languages, including Scala.

What’s so special about Java? Typical “good old” imperative Java code is a finely dispersed mix of statements and expressions, which naturally limits the scope of “type mismatch” highlighting. But that’s no longer the case – here’s a Java 8 version of the above code:


In “functional” code, expressions are no longer separated by statements. This doesn’t go well with the legacy highlighting, which is geared toward “imperative” code. Long underlines are hardly pretty, especially if they span across multiple lines. Outer errors mask inner errors. And if you’re in the process of writing such an expression, it would be constantly underlined solely for being incomplete.

All this poses a problem even in Java, let alone in Scala. Given that typical Scala code is more “functional” than Java code, the problem only gets worse. Because a block is an expression in Scala, arbitrary amounts of code, even imperative code, can be underlined “from tip to toe” if we apply the Java recipe directly:


This problem was encountered as soon as we started to implement type-aware highlighting in the Scala plugin. And so, another recipe was borrowed from Java: to highlight the closing brace:


However, this solution is imperfect, for many reasons. First, a single underlined } char is difficult to notice, especially in large blocks of code. Second, the semantics doesn’t match: in Java, this highlighting indicates that a non-void method is missing a return statement, which has nothing to do with the type mismatch (what’s more, you don’t even “return” values from a block in Scala):


The worst of all, though, is that such a solution is partial – it only addresses one special case without addressing the underlying problem. Not every expression is enclosed in a block:


The scheme could possibly be improved by highlighting the final expression instead of the closing brace. This would make the error easier to spot and would get the semantics right. But, again, this only has to do with blocks, and does nothing otherwise. What’s more, this would interfere with writing new code, as each subsequent expression in a block would be underlined.

A match for type mismatch

To find the solution, let’s look closely at the essence of “type mismatch”:


Does the problem lie in the expression? Sometimes you may need to fix the expression to fix the error. Other times, the problem is somewhere else (which is often the case in Scala because of type inference). As the name suggests, “type mismatch” is about a type, and about a mismatch (more precisely, about the actual type and its mismatch against the expected type). Insightful! But why then do we underline the expression? Well, because we need to underline something, and the type is not visible. Whereas the expected type may be present in the code, the actual type is (almost always) inferred, both in Scala and in Java.

But what if we can highlight a type, not an expression? After all, we already use inlay hints to indicate a missing implicit value instead of underlining the expression. We also use the hints to display implicit conversions – a noticeable improvement over the legacy mode. Thus, we can ascribe the actual type, and then highlight it as we see fit:


This way, we can shift the focus from the expression, which is not necessarily the source of the problem, to the actual cause. Even more importantly, that’s how we can limit the scope of highlighting, regardless of the expression length.

The code of code

Just like with the redesign of implicits, it’s crucial that inlay hints represent valid syntax. This lets people employ the same mental model they use to read code. When a notation is uniform and already familiar, the code is the UI, and even complex highlighting looks simple.

But to achieve that, we need cooperation from the language. For example, since Java syntax lacks named arguments, IntelliJ IDEA has to invent the extraneous notation:


Apparently, this cannot be a valid piece of code. Fortunately, Scala supports named arguments out of the box:


The same goes for type ascription. But is the chosen representation indeed a valid piece of code? Let’s ask the scalac:


This code is not valid in the sense that it doesn’t compile, but no wonder – there’s a type mismatch error to begin with. However, instead of complaining about the syntax, the compiler reports exactly the same error in exactly the same place! Thus, the inlay hint behaves as if it were present in the code “as is”.

Apart from the syntax, there’s the question of style. Originally, parameter name hints look different from code. This is somehow justified in Java, where hints don’t represent code. In Scala, however, hints represent valid code. What’s more, Scala hints behave like code: they provide error highlighting, code folding, tooltips, navigation, and brace matching (and we use the current code style for all of that). When we designed the implicit hints, we unanimously voted to display them “like code”, because this feels more natural. It makes sense to do the same with the type mismatch hints.

It also makes sense to display the existing type annotation hints in the same way, for uniformity. This is also more in line with the implicit hints. As a nice bonus, all the hints then support navigation, folding, tooltips, and brace matching:


The monospaced font is slightly wider, but this is compensated by the added dynamic folding and by showing type hints only when truly needed.

Interestingly, type mismatch hints are not just for type mismatch – they also provide editing assistance. Type ascription for expressions works in the same way as the type annotations for definitions (which often are just named expressions). Both kinds of hints complement the on-demand Type Info action.

Type ascription to type ascription

Showing type ascription as though it was present in the code is great and all, but what if it actually is present? Previously, we highlighted a typed expression as any other expression. That is, we underlined it:


But what are we supposed to do now? Should we ascribe a type twice? While Scala lets you do that – (expression: Actual): Actual, there’s a better way. If you think about it, a type ascription hint is just a blank canvas for error highlighting. If the canvas is already present, so much the better – we can do the highlighting right away:


When some expression lacks an ascribed type, the Scala compiler first has to infer it, and then check for a possible type mismatch. But when a type is already present, the compiler can skip the type inference, and get right to the type checking. That’s what we do now, and this perfectly matches the highlighting of inlay hints (interestingly, this also reduces the scope of type mismatch underlines even without using inlay hints).

So far so good, but what if an ascribed type is incompatible? For example:


Previously, we highlighted the expression as though the type ascription was in fact a type annotation. Should we now ascribe an actual type as (expression: Actual): Incompatible? Not really. There’s a difference between type annotation and type ascription. The former is added to declarations and definitions, and corresponds to explicit types in Java. The latter is added to the expression and corresponds to the cast operator in Java (but that supports only upcasts). In the case of type annotation, the type goes first and the expression follows. By contrast, with type ascription, the expression goes first and the type follows. So, type ascription always specifies the actual type; it’s just that the type might be incompatible. When an expression cannot be upcast to a given type, it’s the type that is the culprit:


That’s how we can reduce the scope of highlighting and keep the highlighting uniform (again, this would work even without using inlay hints).

A diff makes a difference

We’ve done a good job so far. But we need to go deeper narrower. Many Scala types comprise other types and those components are often the original source of a type mismatch. Previously, we compared types as indivisible units:


While technically this wasn’t wrong, it was hardly convenient, because you had to “parse” and “type check” the constituent types in your head to understand why exactly the types do not match. Now, we highlight only the truly “mismatching” parts of a type:


We can even show a fine-grained, vertically-aligned tooltip for a pairwise comparison:


It’s not just a text-based diff either – we take syntactic sugar, subtyping, variance, and other stuff into the account.

The improvement of improvement

That’s the big picture. There are also some special cases, for example, the handling of literal types or method invocations. Although the main work has been completed, the polishing is still in progress. Please use nightly builds for the the most recent updates, and report any bugs or suggestions to YouTrack.

On the whole, it seems that the new highlighting is a substantial improvement over the legacy, “imperative” scheme, and the more “functional” your code, the bigger the improvement. We hope this feature will prove useful for Scala, and we expect other IntelliJ IDEA languages to follow :)

Posted in Uncategorized | 11 Comments

Combine IntelliJ IDEA with Hydra for the Fastest Scala Development Experience

For several years we’ve been working with Triplequote to help them develop the Hydra IntelliJ IDEA plugin. By combining IntelliJ IDEA with the Hydra parallel compiler, you can speed up both Scala development and Scala compilation. Here is a tutorial from the Triplequote team to explain the details.

1. What is Hydra?
2. Install the Hydra IntelliJ IDEA plugin
3. Compile!
4. Low memory detection
5. Compilation bottlenecks detection
6. What next?

What is Hydra?

Hydra, developed by Triplequote, is the only Scala compiler that parallelizes compilation across all available compute cores, speeding up compilation time up to 5x, and providing much faster return-to-productivity.

In addition to faster return to productivity, Hydra ships with built-in compile time monitoring, allowing you to crack open the compiler “black box” with the Hydra Dashboard, providing visualizations and metrics for optimizing your project’s compilation speed over time, and helping teams keep performance under control.

Hydra supports the full language (macros and compiler plugins – Scala.JS included), and it smoothly integrates with your build tool of choice, whether that is sbt, Maven or Gradle.

This article shows how to set up Hydra in IntelliJ IDEA and the many benefits delivered by Hydra in addition to parallel compilation.

Install the Hydra IntelliJ IDEA plugin

To install the Hydra IntelliJ IDEA plugin just search for “Triplequote Hydra” on the Plugins Marketplace.1-installation

Once installed, open the “Hydra Compiler” page and notice that at the top it says “No license detected”.2-license-missing

This is normal, as you need a license to compile with Hydra. Go ahead and hit the button “Get trial license” located on the top-right ([1]), and provide your full name and email. You will immediately receive the license key (check your SPAM folder if you can’t see it).

To use it, go back to the “Hydra Compiler” page, and click the “Enter license key” button ([2]). Paste the Hydra Developer license key you have received and proceed. The “Hydra Compiler” page is updated to confirm the validity of the license.3-license-valid

Congratulations! You are all set and ready to compile your Scala projects with Hydra.


All Scala projects opened in IntelliJ IDEA will now be compiled with Hydra. You don’t need to do anything, it just works!

To try it out, hit “Build > Rebuild project” and notice in the Message view that “N Hydra workers” were used to compile your Scala sources.4-compile-with-hydra

Keep in mind that the default number of Hydra workers used may vary depending on the machine’s hardware, what type of Hydra license is in use, and the number of compiled sources. You can always modify it yourself on the settings page.

Low memory detection

A common (and often ignored) reason for long compilation time is that not enough memory heap is reserved for compilation. In fact, Scala compilation is allocation hungry and can put considerable pressure on the JVM Garbage Collector (GC), to the extreme that more time may be spent GC-ing rather than compiling!

The default heap size reserved for Scala compilation inside IntelliJ IDEA is 1024MB (see the “Scala Compiler Server” preference page under “Preferences” > “Build, Execution, Deployment” > “Compiler” > “Scala Compiler” > “Scala Compile Server”). While this default is good to get started, as modules grows in size and complexity more memory is often needed. Hence, memory usage needs to be constantly monitored during compilation, as otherwise it’s easy to miss that a reason why compilation takes long is because a sizeable amount of time is wasted GC-ing.

The solution to this rather annoying problem is to monitor GC. This can be achieved with the help of a profiler (e.g., VisualVM), but we prefer to automate and when using Hydra to compile there is no need to profile GC, ever. In the event that a considerable amount of time is spent GC-ing during compilation, Hydra reports a warning so that you can act on it.5-gc-overhead

To fix the problem, increase the max heap size in the “Scala Compiler Server” preference page. Be aware that for the change to be picked up the Compiler Server process needs to be restarted – click the throttle icon located at the bottom of the editor, then “Stop” and “Run” again.

Compilation bottlenecks detection

We, as Scala developers, have a tendency to accept our project’s compilation time without ever questioning it. After all, what could we possibly do about it? It turns out there is actually a lot we can do, but to direct our efforts we need to gain visibility into what the compiler does and know where the biggest bangs for the bucks are. Said otherwise, we need to know what Scala sources in our project takes the longest to compile.

One of the greatest features of Hydra is its built-in compile-time monitoring capabilities, and once again Hydra does the heavy lifting for you: if it detects a source file takes more than 5 seconds to compile, it reports the information at the end of the compilation.7-bottleneck-detected

It’s surprising how having this simple piece of information changes entirely the attitude we have in the face of the compiler. All of a sudden, we are surprised that some sources may be taking 5, or 10 seconds, or even more to compile, and hence we dig deeper. How do we do that? And how to know what source(s) takes long to compile? We look at the compilation data collected in the Hydra Dashboard!8-hydra-dashboard

The compilation metrics above were collected while compiling the Doobie open-source project. The Dashboard points out immediately what source took more than 5 seconds to compile.9-slowest-source

This source file contains 250+ case classes definitions. For each case class, the Scala compiler generates several methods in both the class and its companion object. So, while a single case class is very fast to compile, the time adds up when you have over 250+ defined in a single source file. But using case classes was a design decision for the library, so in this case we are ok with the implied compile time cost. This is a remarkable insight we just gained on the project!

Despite the above considerations, there is still value in breaking up a source file that takes a long time to compile, as this reduces the incremental compilation time when editing the specific source. Thanks to IntelliJ IDEA refactoring capabilities, this is easy to achieve, as we can use the “Move” refactoring to split the logic into multiple files, bringing the compilation time of all sources below 5 seconds (the smaller the better!).

As you start exploring the compilation metrics collected on your project, you’ll discover bottlenecks you didn’t expect. A common one is to have imports in scope that are used to generate implicit values through macros (for instance, using the Circe automatic derivation feature). To learn more on this specific topic and how to address it, you may want to watch this ScalaDays talk “5 Things you need to know about Scala compilation” or read the Zalando article “Achieving 3.2x faster compilation time”. The Hydra Dashboard documentation also provides additional useful advice for speeding up compilation further.

What next?

Check out the Hydra documentation to find out more about Hydra and discover how easy it is to integrate it with your build tool of choice. Also, if you would like to benchmark how Hydra performs against the vanilla Scala compiler, you might find the hydraBenchmark command very useful!

And don’t forget to install the Dashboard to quickly identify compilation bottlenecks affecting your Scala projects!

Posted in Uncategorized | 11 Comments

IntelliJ Scala Plugin 2019.1: Highlighting For-comprehensions, Find Usages for Implicits, and More

Some releases are about adding more features. In this release, we’ve focused on improving the existing features of the Scala plugin. These features are the bread and butter of Scala programmers, so we hope the improvements will make our day-to-day work much easier. Here goes:

  1. Highlighting of for-comprehensions
  2. Find Usages for implicits
  3. Decompile to Java for Scala .class files
  4. Customizable Scalafmt version
  5. Separate HOCON plugin
  6. Error highlighting improvements

Highlighting of for-comprehensions

For-comprehensions can sometimes be… incomprehensible. Being a syntactic sugar for composing foreach, map, flatMap, and filter / withFilter operations, for-comprehensions look very differently from the implied method calls. As a Scala programmer, you can probably desugar for-comprehensions in your mind’s eye and, as long as your code compiles, all is well.

The problem begins when there’s an error (yes, once in a while this does happen, you know). An error inside a for-comprehension is actually an error inside its desugared form. Highlighting such an error is tricky, because, in addition to the desugaring, this requires some kind of reverse transformation (“error sugaring”?).

In this release, we present a solution that makes errors in for-comprehensions more comprehensible:

We treat <- as a reference to the corresponding foreach / map / flatMap method, and we treat if as a reference to the filter / withFilter method.

This also lets you invoke GoTo, Quick Definition, and Quick Documentation actions in those spots. For example:

The binding of <- and if symbols to the method calls makes it possible to display implicit arguments:

If that looks a bit unusual, it’s due to the limitation of the language syntax – there’s no way to pass explicit arguments to implicit parameters in a for-comprehension (should we create a SIP?). However, this is consistent with the chosen schema, and is required for showing implicits-related errors:

On top of all that, we’ve improved the desugaring of for-comprehensions. So, if you’re curious about what is behind the curtain of syntactic sugar, you can press Alt + Enter and Desugar for comprehension (or you can rely on the Code | Desugar Scala code action).

Find Usages for implicits

Speaking of implicits, previously the implicits hints allowed you to see usages of implicits and go from usages to definitions, but you couldn’t do the reverse – go from definitions to usages, by invoking Find Usages. Well, now you can:find-usages3

This capability relies on bytecode indices, and can also be used to locate other things that are hidden behind the syntactic sugar, and not present in the source code “as is”, including:

  • apply / unapply method calls,
  • Single Abstract Method (SAM) type instantiations,
  • foreach / map / flatMap / filter / withFilter calls via a for-comprehension (this nicely complements the for-comprehenison highlighting).

The feature goes beyond the typical search. While you can emulate Find Usages by means of a full-text search (albeit imperfectly), you cannot rely on text to find something that is not there. Together with the View | Show Implicit Hints, this feature puts you in control of the ‘invisible’ things.

Decompile to Java for Scala .class files

Did you know that IntelliJ IDEA can decompile .class files to Java? Now this is also possible for compiled Scala files. Suppose you have a third-party class that does something, but you don’t know what – because there are no sources at hand:

That’s where the Decompile to Java may come in handy:

Customizable Scalafmt version

While we supported the scalafmt formatter last year, the Scala plugin could use only a single, bundled version of scalafmt. Now IntelliJ IDEA can automatically download and use whatever version of scalafmt you configure in .scalafmt.conf:

The feature relies on the scalafmt-dynamic module, and so needs an Internet connection. Other than that, this doesn’t require any additional action on your part – everything just works (or at least we hope it does – let us know if it doesn’t for you!).

By the way, we’re also constantly improving the build-in IntelliJ IDEA formatter too :)

Separate HOCON plugin

Hocon is a “Human-Optimized Config Object Notation” format used by many ‘enterprise’ Scala frameworks such as Akka or Play. Because of that, previous versions of the plugin bundled the support of HOCON together with the support of Scala. And yet not everyone who uses Scala needs HOCON.

In this release, we’ve extracted the HOCON support into a separate repository and a separate plugin:

If you do need HOCON, you don’t have to do anything – the HOCON plugin will be automatically installed with the Scala plugin update (and, subsequently, updated). But now you have more flexibility – if you don’t use HOCON, you can easily disable or remove the HOCON plugin in Settings | Plugins.

(Kudos to Roman Janusz for developing the HOCON plugin!)

Error highlighting improvements

While we’re constantly working to make error highlighting even better, sometimes we improve it a lot… like this time! Here are a few highlights:

  • Partial unification and type variable unification in general.
  • Constructor highlighting, and calls to private constructors.
  • better-monadic-for compiler plugin: implicit0 feature.
  • kind-projector: value level polymorphic lambdas.
  • simulacrum: higher-arity type constructors.

Here’s an example to inspire you:

We’re also refactoring our code to make it cleaner and clearer, so if you’ve ever wanted to contribute to the Scala plugin, now is a better time than ever.

While we’ve tried hard to prevent and avoid bugs, absolute perfection is impossible to attain. We’re planning a bugfix release, so please report any issues to YouTrack so that we can fix them as soon as possible.

By the way, the IntelliJ Scala plugin now has an official Twitter account: @IntelliJScala – feel free to follow us for more news and updates.

Develop with Pleasure! No, Drive to Develop! Oh… never mind. Just enjoy!
(and, your feedback is welcome)


The IntelliJ Scala plugin team

Posted in Uncategorized | 3 Comments

How to use the new features of IntelliJ Scala plugin 2018.2

The recently released IntelliJ Scala plugin 2018.2 offers many new features and improvements. Here’s a tutorial on how to use them:

To play with the implicit-related improvements, paste the following code into Project View:

Posted in New Features | Leave a comment

IntelliJ Scala plugin 2018.2: advanced “Implicit” support, improved patterns autocompletion, semantic highlighting, scalafmt and more

First of all, we want to thank all the contributors who have helped us to implement so many useful new features, bug-fixes, and refactorings. You really inspire us to do our very best work. Your input is greatly appreciated!
Let’s take a look at the new features you’ll find in Scala Plugin 2018.2.

Improved display of Implicit conversions / arguments usage

Now the editor is capable of providing much more useful information about the “implicit” things happening in your code:

  • Scala plugin can show implicit conversions and arguments as inline hints.
  • The Editor also shows hints when an implicit argument is used explicitly.
  • Inline hints provide navigation to the implicit value or function declaration.

Note, that sometimes implicit parameters are collapsed to (...). Such folding takes place in case of nested, list of ambiguous arguments, and arguments for conversions.
By the way, ambiguous arguments. We have also re-worked error highlighting. Now, in the same manner, we show places where a function didn’t find an appropriate implicit value, or a set of values is ambiguous.
You’ve probably noticed already, that inline hints work in Editor mode. Enable it by
Ctrl + Alt + Shift + “+” hotkey:

  • A second press of the hotkey expands all foldings.
  • Ctrl + Alt + Shift + “-” will collapse all foldings or disable the mode.

Besides the inline hints we’ve also improved:

  • Parameter Info Tooltip (Ctrl/Cmd + P) by adding the implicit parameter;
  • Implicit Arguments Popup (Ctrl/Cmd + Shift + P) now shows type, structure, and location of arguments.


Autocompletion: exhaustive pattern matching and pattern completion

The Scala plugin now generates an exhaustive match for sealed types with inheritors, Java Enums, and Scala Enumerations. Together with this, the autocompletion list contains an unapply(...) pattern.
Autocompletion in Pattern matching suggests a list of typed patterns for applicable classes and interfaces.

Semantic Highlighting support

Now, you can enable semantic highlighting for your project in Preferences/Settings | Editor | Color Scheme | Scala, and assign distinct colors to a function’s parameter, local variable, variable in a pattern-matching case clause, or variable in a sequence comprehension. Then you can scan through a function to track the variable, with no distracting action isolate one of the variables, or switch focus to another variable.

Scalafmt as an alternative to the built-in formatter

The Scalafmt formatter, which used to be its own standalone plugin, is now part of the Scala plugin. It can be configured at Preferences/Settings | Editor | Code Style | Scala. Once the option is set, the reformat action will invoke scalafmt instead of IntelliJ formatter.
Unlike the IntelliJ built-in formatter, scalafmt hasn’t got a lot of Tabs with detailed options, but instead uses a standard way to pass all the settings by a .conf file. If your project contains a .scalafmt.conf file, the IDE will suggest setting scalafmt as the active formatter. But you can also have a .conf file with another name – just specify it in the “Configuration:” setting.
Sometimes scalafmt may be unable to format invalid code. If you want to see warning notifications in the IDE, check this option.
We have also added the “Reformat on compile” option (enable it in Tab “Other”). The option works for both the IntelliJ built-in and scalafmt formatters.

The release brings lots of other usability enhancements and bug-fixes in Gutter icons, ScalaDoc, and Error highlighting. We would like to say a special thanks to the guys from Wix who improved the Move Refactoring by adding the capability to move object members to another object.
Your feedback is always very welcome. Please do report any bugs you find, and share your ideas, in our issue tracker. Thanks!

Happy developing!

Posted in New Features, Release report | 10 Comments

Ammonite Support

Ammonite, developed by Li Haoyi, is a well-known toolset that lets us use Scala language for scripting purposes. It contains a modernized REPL, a replacement for the Bash system shell, a Filesystem Library, and specific notations for more productive scripting.
Even though many situations in which you would use Ammonite are intended for the command-line mode, quite a few use cases are also relevant for an IDE. After receiving some feedback from our users, we’ve decided to provide advanced support for this technology in our Scala plugin. The set of enhancements includes: support for Ammonite Notations, Run Configuration, a gutter icon for running scripts more easily, and pop-up actions for automatically importing libraries. Read on for more details and screenshots.
Continue reading

Posted in New Features | 13 Comments