AttachMe – Attach the IntelliJ IDEA debugger to forked JVMs automatically

TL;DR

Sometimes you need to attach the IntelliJ IDEA debugger to a running JVM process. Using the Attach to process... action frequently, however, can slow you down a little bit. You might want the IntelliJ IDEA debugger to attach automatically to any process that was spawned outside of IntelliJ IDEA (from the terminal, a build tool, etc.) without having to interact with IntelliJ IDEA each time. Or you might want the debugger to attach to any child JVM that was spawned by the debuggee process. The AttachMe plugin helps solve these issues. Take a look at https://github.com/JetBrains/attachme for instructions and examples.

AttachMe Debugger plugin

The problem

Running a Java application in debug mode is quite straightforward in IntelliJ IDEA. To achieve this, IntelliJ IDEA adds the JDWP JVM agent by adding the -agentlib:jdwp=... option when you run a debug run configuration.

Now imagine a situation where we have to attach the IntelliJ IDEA debugger to a JVM process that’s spawned by a third party process (a build tool for example). First, we need to enable the JDWP agent so that the JVM can attach it. One option is to change the command line arguments and add the -agentib:jdwp=... configuration, but this may prove difficult if we don’t have access to the source code. There are many other ways to achieve this, but one of the easiest is to define an environment variable for the parent process (possibly bash shell), like this:

Next, we can easily attach the IntelliJ IDEA debugger by finding and running the action Attach to process... This seems like an easy solution. However, when we have to do this frequently, it can get a little annoying to tell IntelliJ IDEA to attach to specific processes manually.

The solution

We’ve created a plugin called AttachMe that helps solve this problem. Using this plugin, the IntelliJ IDEA debugger will attach to any JVM application automatically regardless of whether it’s spawned from IntelliJ IDEA, the terminal window, a build tool, etc. As an added bonus, the debugger will attach to all of the subprocesses, essentially giving it the same behavior as set follow-fork-mode child in the GDB debugger.

Attachme in Action

Attachme in Action

Usage

  1. Download and install the plugin from the plugin marketplace. https://plugins.jetbrains.com/plugin/13263-attachme/
  2. Start the AttachMe listener by going to Run > Edit Configurations > Add New. Then search for Attachme, select it, and run.
  3. On the first run, the plugin will install a JVM agent jar and a bash configuration script in the $HOME/.attachme/ directory. To auto-attach the debugger, configure AttachMe like this:

Now you should see a new debugger window attached to the process and any of its child processes.

If you want to have custom JDWP or AttachMe port configuration, you can run it like this:

  1. (Optional) If you want your IntelliJ IDEA run config to use AttachMe, you need to copy and paste the JAVA_TOOL_OPTIONS environment variable to your run configuration. Note that the run config type should be an ordinary run config (not debug), since it will conflict with JDWP args provided by IntelliJ IDEA. To get the value of the JAVA_TOOL_OPTIONS environment variable, just run this in the bash terminal:

How it works

AttachMe makes use of an agent library and the environment variable JAVA_TOOL_OPTIONS to configure the order of agents and to propagate that configuration to child processes. After running source ~/.attachme/conf.sh you can see it by using the following:

This should first load the AttachMe agent, then the JDWP debugger agent. The AttachMe agent tries to identify the listening port number of the debugger (it should always be set to 0, which means any free port). After finding the port, the agent sends it to the IntelliJ AttachMe listener through a TCP connection. It’s really important to have the JDWP bind to port 0 and set suspend to “y”

Known Issues

  • You may have a problem with the bind address, which will manifest itself with this error:
    JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [debugInit.c:750]
    To fix it, try to configure AttachMe with port address 127.0.0.1, like this:

  • AttachMe is not compatible with the JMX agent. If you are having problems running it with the auto generated SpringBoot run configuration, most likely disabling JMX will fix the problem.
Posted in Uncategorized | Leave a comment

Big Data Tools: Experimental Spark Integration, Slack Community, and Feedback

Yesterday we’ve released a fresh update of the Big Data Tools plugin in which we’ve added the integration with AWS S3. With this update, you’ll be able to browse and manage files in your S3 buckets right from the IDE. In yesterday’s announcement we mentioned that there is also another update to the Super-Early-Bird channel. What is this channel? A way the IntelliJ Platform offers to deliver unstable builds with experimental features.

All you have to do to try the experimental features of Big Data Tools, is to register a custom plugin repository URL in the IDE settings. This can be done by clicking the Gear icon in the Plugins settings, and choosing Manage Plugin Repositories

Continue reading

Posted in Big Data Tools, EAP, New Features | Tagged , | 2 Comments

Big Data Tools EAP 4: AWS S3 File Explorer, Bugfixes, and More

The holidays came early this year! Now, when I’ve actually looked at the calendar, I think it’s exactly on time. Whatever the case, we have some presents for you! Just today we’ve released a new update to our Big Data Tools plugin! We hope the update will make your workflow of working with Big Data a bit nicer and more convenient. The major new feature of this update is the integration of AWS S3.

Now, in either the Big Data Tools tool window or the Big Data Tools Connections settings, you can configure an S3 bucket by providing the name of the bucket you’d like to access, the root path (in case you’d like to work with a limited set of files), and your AWS credentials:

Continue reading

Posted in Big Data Tools, EAP, New Features | Tagged , , | 6 Comments

Data Engineering and Developer Tools for Big Data

This is a guest blog post by Jeff Zhang, a speaker at multiple events around Big Data, an active contributor to various open source projects related to Big Data, an Apache member, and a staff engineer at Alibaba Group. Last week, Jeff did a webinar for JetBrains Big Data Tools where he gave an overview on who data engineers are and what tools they use. The webinar was so interesting that we asked Jeff to write this guest post for our blog. This blog post will be especially interesting to you if you’re curious about data engineering, how it’s done, and what tools are used for it. Hopefully, we’ll have more webinars and blog posts on Big Data in the future.

Data engineering is becoming increasingly popular because of the rising interest in big data and AI. Big data creates technical challenges, but it also means there is more value in data. AI drives more data consumption with many applications. In this post, I would like to talk about data engineering and developer tools for big data.

Continue reading

Posted in Big Data Tools, Interviews | Tagged , , | Leave a comment

IntelliJ Scala Plugin 2019.3: Method Chain Hints, Scala REPL, Scala Worksheet, and ScalaTest improvements

Just like IntelliJ IDEA 2019.3, this release of the Scala Plugin is focused primarily on performance boosts and usability enhancements. And yet, we managed to sneak in a cool new feature as well. Let’s take a closer look.

1. Method chain hints
2. Inlay hint settings
3. Searchable settings
4. Scala REPL improvements
5. Scala Worksheet improvements
6. ScalaTest gutter icons
7. Up-to-date Java-to-Scala converter

Method chain hints

To tell the truth, the “new” feature is not entirely new because starting with version 2018.2, IntelliJ IDEA has been able to show type hints for long method chains:

java-method-chain-hints

This functionality, however, has been limited to Java. What’s more, the Java’s approach is hardly applicable in Scala, where long method chains are much more common and we often pass function literals in curly braces, etc. To prevent method chain hints from interfering with Scala code, we can display the hints in a separate column:

scala-method-chain-hints-2

This mode is indeed new, and only the Scala plugin can show the aligned hints so far. In functional programming, there’s the idea that “names are not important, only types are.” Withdrawing judgment, you can definitely learn a lot simply by looking at the type flow. In some sense, this feature reveals the “type level” of your code. (Interestingly, you cannot easily replicate that in the code as such. While you can add type annotations explicitly, there’s no easy way to add type ascription to each line in a long chain of method calls.)

We also offer the classic presentation. Even that mode is less noisy than Java’s counterpart, because the Scala plugin can omit hints when the type is obvious. By the way, you may always tweak the minimum number of unique types in a chain that are required to show the hints.

Inlay hint settings

The Scala plugin now takes advantage of IntelliJ IDEA’s dedicated settings for inlay hints (in Settings | Editor | Inlay hints):

hint-settings

We’ve rearranged and simplified the settings, and added examples that demonstrate the hints right away. The basic options are also available in the context menu:

context-menu

You can open the extended settings by using the Configure… command.

Searchable settings

As you probably know, you can simply type a query to start searching IntelliJ IDEA’s Settings. So far so good, but this capability requires a special assistance from setting providers, and most Scala-related settings have not been searchable… until now:

settings-search

OK, hardly revolutionary, but it’s a nice thing to have.

Scala REPL improvements

Amazing but true: the Scala plugin has a built-in Scala REPL that doesn’t require tinkering with terminals, classpath, and installing Scala binaries in your operating system. You can start the REPL by invoking Tools | Scala REPL (or by pressing Ctrl + Shift + D):

scala-repl

This feature is not widely known, not least because of the previously subpar UX. In this release, we’ve made this feature a lot more usable:

  • It is now called “Scala REPL”, not “Scala Console”.
  • You can use Enter instead of Ctrl (Cmd) + Enter.
  • The input is placed right after the scala> prompt, not a few lines below.

So, the build-in Scala REPL now works very much like the standard Scala REPL. If you did not use the built-in version before, you are welcome to try this one.

Scala Worksheet improvement

Scala Worksheet is another feature that is revamped in this release as we’ve added more reliable error handling. Previously, there could be an error, but no visual feedback. Now, Scala Worksheet can handle multiple types of possible errors, which can now be easily detected and acted upon.

scala-worksheet

ScalaTest gutter icons

You can now run individual ScalaTests:

scala-test-gutter-icons

Up-to-date Java-to-Scala converter

The Java to Scala converter is now up-to-date with Java: it can handle counting loops and switch expressions, and can remove redundant break statements automatically:

java-to-scala-converter-2

And much more!

There are many more improvements, but they are both too numerous and too technical to include in the blog post. For instance, the Scala plugin now supports the so-called “Universal AST”, so that you can benefit from universal Android and Spring inspections in the platform. The support of Scala 2.13 has also been improved.

What really matters is that all this is included in the release. We use the Scala plugin ourselves to develop the Scala plugin, and we feel that spending some time on polishing was definitely worth it – the plugin is now more reliable and pleasant to work with, which is definitely important if you like to develop with pleasure :)

Your feedback is very welcome, as always.

Sincerely,

The IntelliJ Scala Plugin Team

Posted in Uncategorized | 2 Comments

Big Data Tools EAP 3: IntelliJ IDEA 2019.3 EAP Compatibility, Documentation, and More

Hooray!

It’s been a while since our last update on Big Data Tools. Our team has been busy addressing your feedback, fixing bugs, and adding important new features.

First of all, the Big Data Tools plugin is now available in IntelliJ IDEA 2019.3 Beta. In case you don’t know, this is a new upcoming version of the IDE focused entirely on performance and UX. The update brings tons of major and minor improvements overall that will make working with the IDE even more enjoyable.

Second, in order to help you get started with the Big Data Tools plugin, we’ve added the official documentation page to IntelliJ IDEA’s help. In the documentation, you’ll find details on what the plugin is capable of, how to properly configure it, and what to do if you face any difficulties.

To make your experience of working with Zeppelin notebooks even more ergonomic, Zeppelin notebooks are now available in the Recent Files popup.

Using this feature Ctrl+E (Cmd+E for macOS), you can quickly switch to recently edited notebooks – without the need to open the Big Data Tools tool window.

Last but not least, the contents of the recently added Structure popup (which shows the structure of the notebooks) are now displayed in the Structure tool window. Now you can see the notebook and its structure side by side, and then easily navigate to a paragraph.

For a more detailed list of the bugs fixed in this update, please refer to the release notes.

P.S.: If you haven’t already registered for the upcoming live webinar with Jeff Zhang on developer tools for data engineers scheduled for December 3rd, make sure to do so. Hurry, space is limited. If you register, we’ll send you an email with a link to the recording afterwards too (just in case you cannot make it to the live webinar).

P.P.S.: While this update is focused mostly on minor improvements, our team is working hard to bring in some very cool features, which I’m sure lots of data engineers will appreciate. Stay tuned, and spread the word!

The JetBrains Team
The Drive to Develop

Posted in EAP, New Features, Release report | Tagged , , | Leave a comment

Big Data Tools 0.0.356: Minor Improvements and Stability Fixes

Last week we announced the very first version of our new IntelliJ IDEA Ultimate plugin aimed at providing tools for working with Big Data. Currently, the plugin’s functionality is limited to the support for Scala and built-in integration of Zeppelin notebooks. A lot more functionality is to come, including the support for Spark, Hadoop, S3, etc.

Today we’re glad to release a little update (v0.0.356) that brings minor but important bug-fixes and improvements. Below is a list of the changes included:

  • Overall execution progress for Zeppelin notebooks (works only for 0.8.x and higher)

  • The Structure View popup for Zeppelin notebooks

  • In the bottom of the cell, it now displays the author, execution time, and modification time
  • The Basic HTTP authentication issue in Zeppelin connection dialog (it overwrote the Login field, BDIDE-402)
  • The experimental support for Zeppelin 0.9.0 (BDIDE-392)
  • Connection is re-established automatically after a restart of an interpreter
  • The default height of charts is now 10 rows
  • The text output inlays are now not scrolled to the end (as it used to)
  • The issue with processing the results that contain Tab characters (BDIDE-383)

That’s it for now. As we promised, more goodies are on the way. Stay tuned, and in case you haven’t tried the plugin yet, make sure you do. Any feedback is very welcome here in the comments or in our tracker.

P.S. Please join our warm community in Slack to get instant support, tips, updates, and talk to other users of the plugin, Zeppelin, and Spark!

Posted in New Features | Tagged , , | Leave a comment

Meet Big Data Tools – Spark Integration and Zeppelin Notebooks in IntelliJ IDEA

Hooray! Today we have some exciting news for you. After all, it is not often JetBrains introduces new developer tools. Now that you’re excited as well, we’re very pleased to announce Big Data Tools – a new IntelliJ IDEA plugin that integrates Spark and brings support for editing and running Zeppelin notebooks. Now with the Big Data Tools plugin, you can create, edit, and run Zeppelin notebooks without ever having to leave your favorite IDE. The plugin offers smart navigation, code completion, inspections & quick-fixes, and refactoring inside notebooks.

So, what precisely does the plugin support now?

  • Browsing, creating, and deleting notebooks
  • Coding assistance for Scala paragraphs
    • Code completion
    • Rename refactoring
    • Extract variable, method or parameter
    • Go to declaration
    • Show usages
  • Creating and deleting paragraphs
  • Running paragraphs
  • Browsing paragraphs’ output
  • Support for basic visualization

Note coding assistance is currently limited to the Scala code. Other languages will be coming later. Also, in the future, we plan to go beyond Zeppelin notebooks and add more general features that will streamline the developer experience for data engineers, data scientists, and other professionals that work with Big Data.

If you prefer to see it in action rather than just read about it, make sure to watch this short video:

Otherwise, go ahead and try it out for yourself! Below is a short introduction into how to get started. Continue reading

Posted in EAP, New Features | Tagged , , | 8 Comments

Integrating Developer Experiences — The Build Server Protocol in the IntelliJ Scala plugin

IntelliJ IDEA is an IDE – an Integrated Development Environment. I want to focus on the “Integrated” in this post, which is based on my Scala Days talk.

Many developers believe in the “one tool, one job” UNIX philosophy of tooling: A small set of orthogonal, powerful command line tools that individually only do one job, but do it well, and connect through a common interface: stdin/stdout streams. This is great for composing scripts and automating common tasks. But taking a closer look at some of these tools, this breaks down. How often do you have to look up the command line options of find? Myself, it’s almost every time I use it. And each tool has its own idiosyncratic options.

The IDE follows a different approach: one tool, all the jobs. To offer a unified, consistent interface for all the tasks related to software development. Naturally, this leads to big monolithic applications that are hard to embed in an automated pipeline. This often means we need to duplicate some of the configuration around building and running our software in different environments: from the desktop, in a CI pipeline, and finally, in production. But for most of us, this is worth it for the effective developer experience.

Integration rationale

Traditionally IDEs, including IntelliJ IDEA, come with their own versions of a compiler, project configuration, artifact assembly, and so on. That works fine for projects developed by a single programmer, or a small team, but it doesn’t play well with an automated continuous integration and delivery pipeline. We could manually recreate the configuration for the desktop environment, but we both know that it will take about 3 commits for them to get out of sync and cause you a lot of headache trying to figure out where that ClassDefNotFoundError is coming from that is only reproducible on your machine.

The obvious remedy is to write integrations for popular tools and frameworks to import their configuration and map it to the built-in function, taking the responsibility of maintaining the local version off your hands. That works, right? At least, until the next tool or framework you’re eager to try comes along. Which means another IntelliJ IDEA integration. And lots of bug fixing until it works smoothly.

Integrating build tools

Each tool integration is purpose-built on whatever interfaces the tool in question offers. Let’s have a look at the IntelliJ IDEA sbt support. On the frontend side, we have:

  • A menu option to import sbt projects
  • The sbt project toolwindow
  • The sbt preferences pane
  • The sbt shell toolwindow

Not too complicated on the surface. But to make this work, we have introduced a load of background complexity:

  • A special sbt plugin that extracts the project structure and dumps it into an xml file …
  • … that is then parsed by the Scala plugin and transformed into the IntelliJ IDEA project model.
  • Another sbt plugin that modifies the sbt shell output so that IntelliJ IDEA understands when a task is running or has been completed.
  • A hacky mechanism to inject the plugin into your build, so that you don’t have to go through additional configuration steps.
  • Lots of glue code to facilitate the communication between thr IntelliJ IDEA and sbt processes.

Needless to say, this is brittle and breaks left and right. It’s worth it for sbt because of the large user base. But Scala users never seem entirely content with their build tools; someone decides to write a new every year or so. We would need to repeat all the effort we put into supporting sbt for every one of them to provide a comparable experience. How can I justify this for every new tool that, on its own, is only used by a small fraction of our users? On the other hand, the build tools also suffer from a chicken-and-egg problem. Users have come to expect their tools to work together. How are new tools going to get any traction without decent editor or IDE support?

Integral approach

Prior Art

Luckily, there is some prior art on solving this problem. Microsoft’s Visual Studio Code editor popularized the Language Server Protocol (LSP), which addresses the problem of programming language compiler frontends (language servers) talking to editors to provide rich editing features that otherwise would need dedicated editor-side support for a language. While previously, every editor or IDE would need to implement this as language-specific compiler integrations, LSP allows most of the functionality to be implemented in a standardized way. The idea itself is not new, LSP was inspired by other language server approaches such as OmniSharp and the TypeScript Server.

Language servers are also not new to the Scala ecosystem: ENSIME used to be a popular alternative to IDEs.

But LSP has been uniquely successful in terms of adoption. So much so, that people keep asking us if we are going to change the IntelliJ Scala plugin to use it rather than our own implementation of a Scala parser and typechecker, in an effort to address many long-standing highlighting problems. The short answer is probably not, due to various technical issues. There may be space for a hybrid approach, but that is a topic for another post.

Build Server Protocol

At the Scala Center, Jorge Vicente Cantero and Ólafur Páll Geirsson were having a similar problem integrating the various Scala build tools with an editor frontend. Bloop is a build server that aims to optimize compilation times for Scala. It offers a standard implementation of not only the incremental compiler, but also the plumbing of keeping a hot JVM available for compiles and various other optimizations that depend on the structure of the code. At first, bloop was a command line tool that you had to invoke separately from your build tool or editor of choice. As discussed above, this is not ideal from a developer ergonomics point of view.

Naturally, when Jorge proposed the idea of a “Build Server Protocol”, I was very interested. One protocol to build them all? That would solve so many of my problems. So we started collaborating on developing and implementing the protocol. Now, a year after the original announcement, with BSP v2.0 being basically stable and most of the kinks in the implementations in Bloop, Metals and IntelliJ IDEA ironed out, we feel confident to announce it as an officially supported feature in the 2019.2 release of the IntelliJ IDEA Scala plugin.

Implementation

The primary implementations of BSP at this moment are the Scala plugin for IntelliJ IDEA and Metals as clients, and Bloop as server. Server support in other tools is in the works.

IntelliJ Scala as client

The BSP implementation in IntelliJ IDEA allows importing projects from BSP servers and running compilation over the protocol. Unlike previous build tool integrations, this requires no knowledge of the build tool specifics from the IntelliJ IDEA side, nor does it need any IntelliJ-specific support by build tools implementing the protocol.

Since compilation is handled by the build tool, there is no mismatch between the compiler output of IntelliJ IDEA and that of the build tool, even for projects where the project model cannot be successfully imported into the IDE. Note, however, the error highlighting within the editor is still done by the Scala plugin itself, so in some cases there may be red squiggles where there shouldn’t be. We have some ideas on how to improve this by reusing build server messages to inform the highlighting. This would have implications on the features we can offer in the highlighting, so it’s not completely straight-forward.

To use IntelliJ IDEA with Bloop, follow our instructions.

Bloop as server

Bloop is not a full-fledged build tool in itself, but rather a compile server that focuses on compiling, testing and running Scala code. There are plugins for build tools, such as sbt, Mill, Maven, or Gradle, that export the respective project definitions to Bloop config files. This has the benefit of being an easy way to add BSP support to existing tools, even if they don’t have a built-in server. The drawback of this is that importing into the IDE requires a two-step tools-specific process on the user’s side: First, exporting the structure, second, importing it.

But the server nature of Bloop allows another novel use: As a compilation backend to an existing build tool. Instead of writing a tool-specific compiler, usually based on the Zinc incremental compiler library, build tool authors can simply leverage Bloop via BSP. Fury takes this approach.

Further work

Fury is a build tool and dependency manager currently under development by Jon Pretty and VirtusLab. It completely outsources the job of compiling Scala to Bloop, acting as a BSP client. At the same time, it already offers a simple BSP server implementation that allows importing a Fury build into IntelliJ IDEA. With further development, it will support compilation over the protocol by passing through compilation events from Bloop.

Mill is a relatively new build tool by Li Haoyi. It has its own mechanism for exporting builds to IntelliJ IDEA, but this uses an outdated format. It also comes with a module that allows exporting to Bloop, which enables an IntelliJ IDEA import via BSP, at the cost of a two-step process. Neither of these options allow taking advantage of the built-in Mill task graph, which is why we are now working on Mill-native BSP support.

Pants is a build tool by Twitter geared towards large “monorepo” style codebases and designed to build multiple languages. Twitter is currently working on BSP support for Pants.

The Build server protocol is already being used for in-house build tools, allowing a level of IDE integration that would otherwise be hard to achieve and require dedicated plugins.

sbt does not yet incorporate a BSP implementation, but it is possible to build sbt projects over BSP by exporting them to Bloop. In the future, we aim to improve the IntelliJ IDEA sbt support by running it over BSP, which will enable a more robust communication mechanism that the current integrated sbt shell.

Library: bsp4j

Most bsp implementations are based on the bsp4j library, a simple Java library based on lsp4j. The repository comes with a testkit and some tests that help to understand the usage of the library.

Design

The design of the Build Server Protocol closely follows the Language Server Protocol, and reuses some of its data types, such as Diagnostic. This makes it easy to use BSP as a back-end to a language server, or even implement the two protocols side-by-side in a single server.

Like LSP, BSP is a JSON-RPC protocol — a client-server protocol that nonetheless allows bi-directional communication between client and server. Data may be exchanged both by requests and notifications. Requests are typically sent by the client to request some data about the build, or ask the server to execute an action, such as compile or test. Notifications are more often sent by the server to give updates about the current state of a task, or report compilation problems (diagnostics).

Concepts

I’m going to give a brief overview of some of the main BSP concepts to give you an idea of how it works and an orientation for implementation, as well as considerations for the design. If you are interested implementing BSP, please refer to the full BSP specification.

Server discovery

connection-protocol

Before talking to each other, the IDE and build server need to know how to find each other. In LSP, this is usually handled by a language-specific client implementation that knows how to start the right language server. This was not quite good enough for our purposes. We wanted any client to be able to connect to any server and offer basic functionality out of the box, without requiring a dedicated plugin.

Our solution is the BSP connection protocol. It’s quite simple:

  1. The server places a JSON connection file

    into the {workspace}/.bsp/ or a OS-dependent system or user directory:

  2. The client discovers this file, and starts the server process.
  3. Client and server write and read JSON-RPC messages over stdin/stdout of the server process.

This very simple cross-platform mechanism enables a zero-configuration approach: no pipes, ports, or IPs necessary. Of course the server process is free to implement a more advanced protocol behind the scenes, including remote communication. But the client doesn’t need to know about it.

Build structure

buildtargets

A major part of the Build Server Protocol is to standardize the representation of the build graph. Most build tools have some way of modeling nodes in a build graph in term of components: sbt calls them projects, IntelliJ names them modules, in Bazel they’re targets. Depending on the tool, these nodes may be a different level of granularity, and don’t always map to another tool’s idea of a node. Nonetheless, they all share some commonalities that BSP unifies under the concept of targets, closely following the Bazel design:

  • Targets may depend on one another, forming a directed acyclic graph.
  • A target contains some metadata about the language and language level it is compiled with, and possible actions on this target (such as compile or test).
  • A target can be associated with any number of source files or directories
  • Sources associated within a target are compiled as a unit.

The build targets in a workspace are requested in a single workspace/buildTargets request from the client to the server, while the associated sources are requested separately in the buildTarget/sources request.

In BSP, a set of sources may be associated with any number of targets. This is in contrast to the IntelliJ IDEA project model, where a source file may only be part of a single module. This limitation exists to allow the IDE to always know how a file needs to be highlighted, and is tightly coupled to the implementation of many of IntelliJ IDEA’s internals. This has always been a challenge when importing sbt projects that cross-compile sources to several targets (typically scala.js or Scala Native), as there is no straightforward mapping of shared sources to the IntelliJ IDEA model. Our workaround is to create synthetic “shared” modules that are in turn dependencies of the actual modules sharing the sources. This works at least some of the time, but will often have unexpected results. We are looking at better ways to support shared sources in the future.

module-mapping_upd

Actions

The other major benefit of a build protocol is executing typical build server actions in a standard way. The most common one of course is compilation. Without any tool integration, this is where you’d switch to the terminal to type a command whenever you want to know about all the programming errors you’ve made. But what you’d rather do, is to just press a button in the IDE, or let it happen every time you save a file. To this end, BSP offers some requests.

The buildTarget/compile request is sent from the client to the server with parameters for which targets the server should perform a compilation. Once completed, the server will reply with a CompileResult that contains a status code about the compilation result. On its own, this isn’t very informative, so during the compilation, the server may send any number of diagnostics and progress notifications (see below)

Analogously, buildTarget/test requests running tests within some modules, and will return with a TestResult. While the request is being run, any number of test-specific progress notifications may be sent from the client to the server.

Diagnostics

compileRequest

As in LSP, diagnostics give information about issues at certain code points — compiler warnings and errors. Diagnostics may be sent at any time from the server to the client as build/publishDiagnostics notification, but usually they will be associated with a compile request.

A point to note: One diagnostics notification always contains all the individual diagnostics that are currently valid for a single file. Subsequent notifications for the same file invalidate the previous ones. This allows the editor to persistently highlight all the currently valid errors, and clear them on the next notification if they are fixed.

Progress

Often tasks in a build server will be long-running and involve many sub-tasks. We added task progress notifications to the protocol to let the client give informative feedback on the state of a build.

tasks

Task notifications are sent from the server to the client to give updates on the running tasks. Tasks notifications can be one of three types:

  • build/taskStart signals the beginning of some task, such as compilation of a target or running of a test.
  • build/taskProgress gives updates on how far the task has progressed.
  • build/taskFinish signals the completion of a task.

When a server implements these notifications, running a build will look like this in IntelliJ IDEA:

buildtoolwindow

Towards an Integrated Developer Experience

The Build Server Protocol implementation in the Scala plugin for IntelliJ IDEA helps you to be productive right now with Bloop, or to get hacking on Fury. So far the implementation is still limited to Scala targets. There’s still plenty of work to be done to create a fully Integrated Developer Experience.

Looking at only Scala projects, some of the implementation is still incomplete, but it is improving. We are currently working on running tests over BSP In IntelliJ IDEA, which will make it easier for test tool authors to offer IDE support.

Building on the base protocol, it will be easier to offer robust language- and tool-specific support for sbt, Mill, and others in the future. This could include editor support for build files, or running arbitrary tasks.

Once the protocol is established in the Scala ecosystem, we want to take a closer look at how it can help with polyglot tools like Bazel and Pants as well as within other language ecosystems.

Get involved!

We’d love for you to get involved and help us to improve the experience you have with build tool and editor support! The simplest way is to try it for yourself and give us feedback.

If you want to get your hands dirty, work directly on the client implementation in IntelliJ Scala or contribute to existing servers, or create new implementations in Bloop, Mill, Fury, sbt, or others and chat on the BSP Gitter channel

Links

Posted in New Features | Tagged , | 1 Comment

IntelliJ Scala Plugin 2019.2: Functional Code Highlighting, Fine-grained Type Diff, Case Clause Completion, Build Server Protocol, and More

The 2019.2 release of the Scala plugin offers many improvements in different areas. The key focus of this release is a new code highlighting paradigm whereby the highlighting is now cleaner, yet provides even more coding assistance.

1. Functional code highlighting
2. Fine-grained type diff
3. Interactive type hints
4. Wrap / unwrap expressions as you type
5. Unobtrusive redundant code inspections
6. Case clause completion
7. Build Server Protocol
8. Language injection for multiline strings
9. Duplicate code fragment inspection

Functional code highlighting

Previously, IntelliJ IDEA highlighted type mismatch errors with a red squiggly underline. But long underlines are hardly pretty, especially if they span across multiple lines. Outer errors mask inner errors. And if you’re in the process of writing such an expression, it would be constantly underlined solely for being incomplete.

Now we can do better: instead of underlining the whole expression, the Scala plugin now shows a type ascription hint and highlights only the part that doesn’t match the expected type:
ScalaTypeMismatchHints

This significantly cleans up your editor, while providing even more coding assistance. To learn about the “whys” and the “hows”, see our recent post, Functional Highlighting for Functional Programming. (Note that the mode is controlled by Settings | Languages | Scala | Type mismatch highlighting.)

Fine-grained type diff

Previously, we compared types as indivisible units. While technically this wasn’t wrong, it was hardly convenient, because you had to “parse” and “type-check” the constituent types in your head to understand why exactly the types did not match.

Now we can show a fine-grained, vertically-aligned tooltip for a pairwise comparison:
ScalaDiffTooltip

It’s not just a text-based diff – the algorithm takes syntactic sugar, subtyping, variance, and other tricky stuff into the account.

Interactive type hints

Type annotation hints now support dynamic folding, tooltips, navigation, and brace matching:
ScalaInteractiveTypeHints

What’s more, type hints are shown only when they are truly needed (so, if you previously disabled type hints because they were too noisy, consider giving them another try). That’s the second editor cleanup, alongside with the new type mismatch highlighting.

Wrap / unwrap expressions as you type

Before, you had to invoke the Add braces around single line intention to wrap an expression in curly braces, or the Remove braces intention to unwrap an expression.

Now, IntelliJ IDEA can do that automatically, as you type:

While the intentions are still available (and work just fine), the automatic balancing is much more discoverable and is very handy in practice.

Unobtrusive redundant code inspections

IntelliJ IDEA has always highlighted unused imports or unreachable code as “unused” (as defined by the Color Scheme). Other things, however, were highlighted as “generic warning or error”, which polluted code and concealed actual errors.

Now we highlight all redundant code uniformly, so that the highlighting looks unobtrusive:
ScalaUnusedCodeInspections

That’s the third editor cleanup. Since the highlighting now doesn’t get in your way, we can implement many more useful related inspections. Feel free to vote for the ones you like the most!

Case clause completion

The Scala plugin has already offered exhaustive match completion for quite some time.

Now you can also complete individual case clauses:

What’s more, the completion now works in partial functions, as well as in match expressions.

Build Server Protocol

The Build Server Protocol (BSP) defines a standard way for build tools to talk to IDEs, allowing you to open projects, run compilation, perform tasks, display error messages, and monitor progress updates directly in the IDE.

For a while, BSP support was an experimental feature, but not anymore – now it’s fully supported:
ScalaBSP

The supported build tools include Bloop and Fury so far. Support for Mill is coming soon. We’re going to post more details on the BSP integration soon, so stay tuned.

Language injection into multiline strings

Now let’s talk about a few features that are only available in IntelliJ IDEA Ultimate. The first one is the support of language injection into multiline strings:
ScalaLanguageInjection

This feature may come handy for embedding SQL, HTML, or JavaScript into Scala code.

Duplicated code fragment inspection

Did you read the HTML comment in the previous release announcement? If you didn’t, here it is again: “We’ve also implemented duplicates detection, but this capability is not yet enabled in the IntelliJ Platform; expect that feature in the next release”. So, here goes:
duplicate-code-fragment

Because the feature is implemented as an inspection, you can invoke Run inspection by name | Whole project to fix all the duplicates at once. The algorithm can filter out whitespaces, syntactic sugar, and other accidental details. So, don’t repeat yourself, don’t repeat yourself.

Summary

The new features work in synergy with the implicit hints, which have already cleaned the editor from pervasive “implicit underlines”. The Scala plugin now offers a unique combination of clarity and power: your code looks almost as clean and unclouded as in a plain text editor, while you have intelligence that even the Scala compiler can’t provide, including implicit hints, fine-grained type diffs, inspections, and much more.

Your feedback is very welcome as always. Please report any bugs to YouTrack so that we can fix them as soon as possible. Thank you!

Sincerely,

The IntelliJ Scala plugin team

Posted in New Features, Release report | 8 Comments