Integrating Developer Experiences — The Build Server Protocol in the IntelliJ Scala plugin

Justin Kaeser

IntelliJ IDEA is an IDE – an Integrated Development Environment. I want to focus on the “Integrated” in this post, which is based on my Scala Days talk.

Many developers believe in the “one tool, one job” UNIX philosophy of tooling: A small set of orthogonal, powerful command line tools that individually only do one job, but do it well, and connect through a common interface: stdin/stdout streams. This is great for composing scripts and automating common tasks. But taking a closer look at some of these tools, this breaks down. How often do you have to look up the command line options of find? Myself, it’s almost every time I use it. And each tool has its own idiosyncratic options.

The IDE follows a different approach: one tool, all the jobs. To offer a unified, consistent interface for all the tasks related to software development. Naturally, this leads to big monolithic applications that are hard to embed in an automated pipeline. This often means we need to duplicate some of the configuration around building and running our software in different environments: from the desktop, in a CI pipeline, and finally, in production. But for most of us, this is worth it for the effective developer experience.

Integration rationale

Traditionally IDEs, including IntelliJ IDEA, come with their own versions of a compiler, project configuration, artifact assembly, and so on. That works fine for projects developed by a single programmer, or a small team, but it doesn’t play well with an automated continuous integration and delivery pipeline. We could manually recreate the configuration for the desktop environment, but we both know that it will take about 3 commits for them to get out of sync and cause you a lot of headache trying to figure out where that ClassDefNotFoundError is coming from that is only reproducible on your machine.

The obvious remedy is to write integrations for popular tools and frameworks to import their configuration and map it to the built-in function, taking the responsibility of maintaining the local version off your hands. That works, right? At least, until the next tool or framework you’re eager to try comes along. Which means another IntelliJ IDEA integration. And lots of bug fixing until it works smoothly.

Integrating build tools

Each tool integration is purpose-built on whatever interfaces the tool in question offers. Let’s have a look at the IntelliJ IDEA sbt support. On the frontend side, we have:

• A menu option to import sbt projects
• The sbt project toolwindow
• The sbt preferences pane
• The sbt shell toolwindow

Not too complicated on the surface. But to make this work, we have introduced a load of background complexity:

• A special sbt plugin that extracts the project structure and dumps it into an xml file …
• … that is then parsed by the Scala plugin and transformed into the IntelliJ IDEA project model.
• Another sbt plugin that modifies the sbt shell output so that IntelliJ IDEA understands when a task is running or has been completed.
• A hacky mechanism to inject the plugin into your build, so that you don’t have to go through additional configuration steps.
• Lots of glue code to facilitate the communication between thr IntelliJ IDEA and sbt processes.

Needless to say, this is brittle and breaks left and right. It’s worth it for sbt because of the large user base. But Scala users never seem entirely content with their build tools; someone decides to write a new every year or so. We would need to repeat all the effort we put into supporting sbt for every one of them to provide a comparable experience. How can I justify this for every new tool that, on its own, is only used by a small fraction of our users? On the other hand, the build tools also suffer from a chicken-and-egg problem. Users have come to expect their tools to work together. How are new tools going to get any traction without decent editor or IDE support?

Integral approach

Prior Art

Luckily, there is some prior art on solving this problem. Microsoft’s Visual Studio Code editor popularized the Language Server Protocol (LSP), which addresses the problem of programming language compiler frontends (language servers) talking to editors to provide rich editing features that otherwise would need dedicated editor-side support for a language. While previously, every editor or IDE would need to implement this as language-specific compiler integrations, LSP allows most of the functionality to be implemented in a standardized way. The idea itself is not new, LSP was inspired by other language server approaches such as OmniSharp and the TypeScript Server.

Language servers are also not new to the Scala ecosystem: ENSIME used to be a popular alternative to IDEs.

But LSP has been uniquely successful in terms of adoption. So much so, that people keep asking us if we are going to change the IntelliJ Scala plugin to use it rather than our own implementation of a Scala parser and typechecker, in an effort to address many long-standing highlighting problems. The short answer is probably not, due to various technical issues. There may be space for a hybrid approach, but that is a topic for another post.

Build Server Protocol

At the Scala Center, Jorge Vicente Cantero and Ólafur Páll Geirsson were having a similar problem integrating the various Scala build tools with an editor frontend. Bloop is a build server that aims to optimize compilation times for Scala. It offers a standard implementation of not only the incremental compiler, but also the plumbing of keeping a hot JVM available for compiles and various other optimizations that depend on the structure of the code. At first, bloop was a command line tool that you had to invoke separately from your build tool or editor of choice. As discussed above, this is not ideal from a developer ergonomics point of view.

Naturally, when Jorge proposed the idea of a “Build Server Protocol”, I was very interested. One protocol to build them all? That would solve so many of my problems. So we started collaborating on developing and implementing the protocol. Now, a year after the original announcement, with BSP v2.0 being basically stable and most of the kinks in the implementations in Bloop, Metals and IntelliJ IDEA ironed out, we feel confident to announce it as an officially supported feature in the 2019.2 release of the IntelliJ IDEA Scala plugin.

Implementation

The primary implementations of BSP at this moment are the Scala plugin for IntelliJ IDEA and Metals as clients, and Bloop as server. Server support in other tools is in the works.

IntelliJ Scala as client

The BSP implementation in IntelliJ IDEA allows importing projects from BSP servers and running compilation over the protocol. Unlike previous build tool integrations, this requires no knowledge of the build tool specifics from the IntelliJ IDEA side, nor does it need any IntelliJ-specific support by build tools implementing the protocol.

Since compilation is handled by the build tool, there is no mismatch between the compiler output of IntelliJ IDEA and that of the build tool, even for projects where the project model cannot be successfully imported into the IDE. Note, however, the error highlighting within the editor is still done by the Scala plugin itself, so in some cases there may be red squiggles where there shouldn’t be. We have some ideas on how to improve this by reusing build server messages to inform the highlighting. This would have implications on the features we can offer in the highlighting, so it’s not completely straight-forward.

To use IntelliJ IDEA with Bloop, follow our instructions.

Bloop as server

Bloop is not a full-fledged build tool in itself, but rather a compile server that focuses on compiling, testing and running Scala code. There are plugins for build tools, such as sbt, Mill, Maven, or Gradle, that export the respective project definitions to Bloop config files. This has the benefit of being an easy way to add BSP support to existing tools, even if they don’t have a built-in server. The drawback of this is that importing into the IDE requires a two-step tools-specific process on the user’s side: First, exporting the structure, second, importing it.

But the server nature of Bloop allows another novel use: As a compilation backend to an existing build tool. Instead of writing a tool-specific compiler, usually based on the Zinc incremental compiler library, build tool authors can simply leverage Bloop via BSP. Fury takes this approach.

Further work

Fury is a build tool and dependency manager currently under development by Jon Pretty and VirtusLab. It completely outsources the job of compiling Scala to Bloop, acting as a BSP client. At the same time, it already offers a simple BSP server implementation that allows importing a Fury build into IntelliJ IDEA. With further development, it will support compilation over the protocol by passing through compilation events from Bloop.

Mill is a relatively new build tool by Li Haoyi. It has its own mechanism for exporting builds to IntelliJ IDEA, but this uses an outdated format. It also comes with a module that allows exporting to Bloop, which enables an IntelliJ IDEA import via BSP, at the cost of a two-step process. Neither of these options allow taking advantage of the built-in Mill task graph, which is why we are now working on Mill-native BSP support.

Pants is a build tool by Twitter geared towards large “monorepo” style codebases and designed to build multiple languages. Twitter is currently working on BSP support for Pants.

The Build server protocol is already being used for in-house build tools, allowing a level of IDE integration that would otherwise be hard to achieve and require dedicated plugins.

sbt does not yet incorporate a BSP implementation, but it is possible to build sbt projects over BSP by exporting them to Bloop. In the future, we aim to improve the IntelliJ IDEA sbt support by running it over BSP, which will enable a more robust communication mechanism that the current integrated sbt shell.

Library: bsp4j

Most bsp implementations are based on the bsp4j library, a simple Java library based on lsp4j. The repository comes with a testkit and some tests that help to understand the usage of the library.

Design

The design of the Build Server Protocol closely follows the Language Server Protocol, and reuses some of its data types, such as Diagnostic. This makes it easy to use BSP as a back-end to a language server, or even implement the two protocols side-by-side in a single server.

Like LSP, BSP is a JSON-RPC protocol — a client-server protocol that nonetheless allows bi-directional communication between client and server. Data may be exchanged both by requests and notifications. Requests are typically sent by the client to request some data about the build, or ask the server to execute an action, such as compile or test. Notifications are more often sent by the server to give updates about the current state of a task, or report compilation problems (diagnostics).

Concepts

I’m going to give a brief overview of some of the main BSP concepts to give you an idea of how it works and an orientation for implementation, as well as considerations for the design. If you are interested implementing BSP, please refer to the full BSP specification.

Server discovery

Before talking to each other, the IDE and build server need to know how to find each other. In LSP, this is usually handled by a language-specific client implementation that knows how to start the right language server. This was not quite good enough for our purposes. We wanted any client to be able to connect to any server and offer basic functionality out of the box, without requiring a dedicated plugin.

Our solution is the BSP connection protocol. It’s quite simple:

1. The server places a JSON connection file

   into the {workspace}/.bsp/ or a OS-dependent system or user directory:

     {
       "name": "My Build Tool",
       "version": "21.3",
       "bspVersion": "2.0.0",
       "languages": ["scala", "javascript", "rust"],
       "argv": ["my-build-tool", "bsp"]
     }
   

2. The client discovers this file, and starts the server process.
3. Client and server write and read JSON-RPC messages over stdin/stdout of the server process.

This very simple cross-platform mechanism enables a zero-configuration approach: no pipes, ports, or IPs necessary. Of course the server process is free to implement a more advanced protocol behind the scenes, including remote communication. But the client doesn’t need to know about it.

Build structure

A major part of the Build Server Protocol is to standardize the representation of the build graph. Most build tools have some way of modeling nodes in a build graph in term of components: sbt calls them projects, IntelliJ names them modules, in Bazel they’re targets. Depending on the tool, these nodes may be a different level of granularity, and don’t always map to another tool’s idea of a node. Nonetheless, they all share some commonalities that BSP unifies under the concept of targets, closely following the Bazel design:

• Targets may depend on one another, forming a directed acyclic graph.
• A target contains some metadata about the language and language level it is compiled with, and possible actions on this target (such as compile or test).
• A target can be associated with any number of source files or directories
• Sources associated within a target are compiled as a unit.

The build targets in a workspace are requested in a single workspace/buildTargets request from the client to the server, while the associated sources are requested separately in the buildTarget/sources request.

In BSP, a set of sources may be associated with any number of targets. This is in contrast to the IntelliJ IDEA project model, where a source file may only be part of a single module. This limitation exists to allow the IDE to always know how a file needs to be highlighted, and is tightly coupled to the implementation of many of IntelliJ IDEA’s internals. This has always been a challenge when importing sbt projects that cross-compile sources to several targets (typically scala.js or Scala Native), as there is no straightforward mapping of shared sources to the IntelliJ IDEA model. Our workaround is to create synthetic “shared” modules that are in turn dependencies of the actual modules sharing the sources. This works at least some of the time, but will often have unexpected results. We are looking at better ways to support shared sources in the future.

Actions

The other major benefit of a build protocol is executing typical build server actions in a standard way. The most common one of course is compilation. Without any tool integration, this is where you’d switch to the terminal to type a command whenever you want to know about all the programming errors you’ve made. But what you’d rather do, is to just press a button in the IDE, or let it happen every time you save a file. To this end, BSP offers some requests.

The buildTarget/compile request is sent from the client to the server with parameters for which targets the server should perform a compilation. Once completed, the server will reply with a CompileResult that contains a status code about the compilation result. On its own, this isn’t very informative, so during the compilation, the server may send any number of diagnostics and progress notifications (see below)

Analogously, buildTarget/test requests running tests within some modules, and will return with a TestResult. While the request is being run, any number of test-specific progress notifications may be sent from the client to the server.

Diagnostics

As in LSP, diagnostics give information about issues at certain code points — compiler warnings and errors. Diagnostics may be sent at any time from the server to the client as build/publishDiagnostics notification, but usually they will be associated with a compile request.

A point to note: One diagnostics notification always contains all the individual diagnostics that are currently valid for a single file. Subsequent notifications for the same file invalidate the previous ones. This allows the editor to persistently highlight all the currently valid errors, and clear them on the next notification if they are fixed.

trait PublishDiagnosticsParams {
  /** The document where the diagnostics are published. */
  def textDocument: TextDocumentIdentifier

  /** The build target where the diagnostics origin.
  * It is valid for one text document to belong to multiple
  * build targets, for example sources that are compiled against multiple
  * platforms (JVM, JavaScript). */
  def buildTarget: BuildTargetIdentifier

  /** The request id that originated this notification. */
  def originId: Option[String]

  /** The diagnostics to be published by the client. */
  def diagnostics: List[Diagnostic]
}

Progress

Often tasks in a build server will be long-running and involve many sub-tasks. We added task progress notifications to the protocol to let the client give informative feedback on the state of a build.

Task notifications are sent from the server to the client to give updates on the running tasks. Tasks notifications can be one of three types:

• build/taskStart signals the beginning of some task, such as compilation of a target or running of a test.
• build/taskProgress gives updates on how far the task has progressed.
• build/taskFinish signals the completion of a task.

When a server implements these notifications, running a build will look like this in IntelliJ IDEA:

Towards an Integrated Developer Experience

The Build Server Protocol implementation in the Scala plugin for IntelliJ IDEA helps you to be productive right now with Bloop, or to get hacking on Fury. So far the implementation is still limited to Scala targets. There’s still plenty of work to be done to create a fully Integrated Developer Experience.

Looking at only Scala projects, some of the implementation is still incomplete, but it is improving. We are currently working on running tests over BSP In IntelliJ IDEA, which will make it easier for test tool authors to offer IDE support.

Building on the base protocol, it will be easier to offer robust language- and tool-specific support for sbt, Mill, and others in the future. This could include editor support for build files, or running arbitrary tasks.

Once the protocol is established in the Scala ecosystem, we want to take a closer look at how it can help with polyglot tools like Bazel and Pants as well as within other language ecosystems.

Get involved!

We’d love for you to get involved and help us to improve the experience you have with build tool and editor support! The simplest way is to try it for yourself and give us feedback.

If you want to get your hands dirty, work directly on the client implementation in IntelliJ Scala or contribute to existing servers, or create new implementations in Bloop, Mill, Fury, sbt, or others and chat on the BSP Gitter channel

Links

• BSP library implementations (bsp4j/bsp4s) and testkit
• BSP specification
• Integrating Developer Experiences talk at Scala Days 2019
• Build Server Protocol and new IDEAs talk at ScalaSphere 2018
• BSP 1.0 announcement
• BSP Gitter chat