Fleet Below Deck, Part V – The Story of Code Completion
This is a multipart series on building Fleet, a next-generation IDE by JetBrains.
- Part I – Architecture Overview
- Part II – Breaking Down the Editor
- Part III – State Management
- Part IV – Distributed Transactions
- Part V – The Story of Code Completion
- Part VI – UI with Noria
In Parts III and IV of this series, we discussed the complicated abstract architectural concepts involved in state management and how they are synchronized across Fleet’s distributed components. Now it’s time for us to look at a more familiar function – Code completion – and see how it is implemented in Fleet.
Fleet is not a tremendous breakthrough in the world of code completion, but its architecture and distributed nature made their mark here and there. Imagine a typical Fleet user, blessed (or doomed) to work with several programming languages. This type of software developer struggles with Go and JavaScript, adores Kotlin and Rust, and writes notes in text files. Is there a code completion solution to cover all of these situations? Let’s find out.
Code Completion as seen by end users
Suppose we start by making some text notes. Let’s look at the code completion we have so far:
This behavior is built into Fleet’s editor: it looks into the document and considers which words are good candidates for code completion. This sounds somewhat reasonable, although you would imagine that some machine learning magic takes place here, based on the content of your other previously taken notes.
Anyway, let’s move on to some Kotlin programming.
This completion behavior is not particularly impressive. Basically it boils down to the same algorithm as in the case of text notes. Fleet helps us out here by suggesting that we enable smart mode. Let’s do that!
This looks better: We’ve been given some accurate suggestions based on the code analysis. The question is, what is responsible for providing these completion items? It’s definitely not the editor, because we need a lot of information about the code itself and the libraries it might potentially use.
Note that completion is not necessarily appending code. In fact, it’s changing code. In the following example, code completion results in removing the dot character and adding enclosing braces:
It’s also possible to use code completion to generate common code fragments from predefined snippets. In the example below, we write Rust with smart mode enabled. Once we type for and press Ctrl-Space, the following snippet is inserted:
The completion result is some template text with several placeholders denoted by carets. We can move to the next one with the TAB key. Inserting snippets is quite a tedious task. Fleet doesn’t have to analyze the code to do that. This feature can be easily implemented in the editor itself. By exploring these examples, we might see that completion results rely heavily on smart mode, though even without smart mode, we’ve got something to work with. We can also see that we may have some basic code appending as well as code refactoring or inserting complex code fragments with placeholders. Now it’s time to see what’s going on below deck.
What’s going on inside
The following diagram gives a pretty good overview of how code completion in Fleet works.
In some cases, Fleet’s frontend can provide code completion itself. For example, there are several predefined JSON schemes for completing, say, settings.json, a file with Fleet’s settings. If we enable smart mode, we may start observing the whole universe of completion engines powered by either IntelliJ IDEA, or ReSharper, or myriad LSP (language server protocol) servers. Rust-analyzer was the first LSP server supported in Fleet. As the Fleet project is progressing nicely, there will be others in the future.
IntelliJ IDEA backend and LSP servers provide code completion as part of their services. They provide many other services ranging from maintaining indentation rules to reporting programming errors in your code. For some programming languages, having more than one backend engine is fine. In the future, Fleet’s users will be able to choose whichever engine they prefer.
Code completion can be broken down into several components:
- Completion items available and reasonable for some particular source position.
- A completion service as an engine responsible for providing completion items.
- A completion API that unifies different completion services.
- A completion session that is responsible for delivering completion items to users and applying the one that is chosen.
These deserve to be discussed a bit more deeply.
Completion Items
A completion item refers to one available option for proceeding with the current source position.
There are three kinds of completion items in Fleet:
- A basic completion item is simply some text that should be appended to the current cursor position.
- A snippet is a template with placeholders. Once inserted into the source file, snippets support navigation between placeholders and dynamic suggestion of completion items based on what was entered in the other placeholders. These snippets provide the same functionality as the Live templates in IntelliJ IDEA.
- Declarative Insert is a collection of editing instructions (like insert, remove, replace) that should be applied to code to finalize a code completion action.
The specific kind of completion item defines a few things: what is shown in the completion item list, what is done to the code after applying it, what is a priority of some particular code suggestion, and so on. Applying completion items can be more or less trivial depending on their kind. If completion requires some sophisticated code manipulation that is not expressible in a declarative way, Fleet has to ask the backend to execute completion on the backend’s side, not in Fleet’s editor.
The procedure of applying a completion item doesn’t result in changing document content. Instead, it produces a set of operations that should be applied to that content. This is where we arrive at the first bridge to Fleet’s state management. These text manipulating operations will then be applied with Fleet’s distributed nature in mind.
Completion items don’t appear out of nowhere. There must be some service that generates them based on the current source code location and the current document version. Since there could be more than one such service, we need an API to get unified access to all of them.
Completion API and Completion Services
Fleet’s Completion API is thin and allows us:
- To get completion items for the given source code location.
- To complete a current statement, if there is only one sensible way to do that.
- To apply a completion for us, if there is something nontrivial to do.
- To clean up resources used by the completion engine.
How does Fleet know which completion services are available? Are discovery services used? Those of you who have read the previous blog posts may realize that we can in fact use our state management engine for that. Any completion service is just an entity loaded into Fleet’s state. In fact, any functionality in Fleet often comes in the form of some entity.
These service entities know which document types they support. Fleet looks up the state and picks the first one responsible for the current document type. If nothing is found, Fleet falls back on predefined snippets or document words list, as seen in the diagram above. By loading and unloading these entities (read: enabling and disabling corresponding plugins), end users have control over what is used to complete their code.
As of now, there are two primary completion services in Fleet:
- One is responsible for getting completion items from the IntelliJ IDEA backend.
- The other gets completion items from the LSP connection.
Both of them implement the Completion API so Fleet can use them in a uniform manner. We won’t discuss here how those services actually produce completion items – this is of no interest to Fleet. Fleet is just a simple text editor at the end of the day!
Completion Session
Performing code completion is a timely process. Once it is initiated by the user, Fleet has to find an appropriate completion service and fire it up. The received items have to be displayed in a popup window. Once the user has made their choice, the chosen item has to be applied everywhere (in the workspace and all the frontends).
In fact, this process is usually even more complicated than that. First, coming up with good quality completion items takes time and it would be unwise to wait until they’re all ready. Instead, they are provided as a flow of items. As soon as Fleet gets a new portion of items, the component responsible for displaying them has to be updated.
Second, the user may continue typing after requesting code completion. Restarting completion actions from scratch would be too inefficient. Calling a backend for every typed character would also be quite expensive. Fleet can update the list of already received completion items by filtering and re-ranking them. Re-ranking is quite tricky. It’s implemented via several additional priority characteristics that are able to reflect prefix-matching.
Ultimately, the user can cancel the completion if they’re unhappy with the suggestions.
The completion session is yet another entity in Fleet’s state that is responsible for managing all that has been mentioned above and more. For example, it deals with source code positions that may very well be outdated, given that other users might be editing the same document simultaneously using another Fleet frontend!
Sometimes it’s hard to believe that all of this actually works. Take another look at the screenshots in the first section and pay attention to the UI details accompanying the code completion session. Warnings and errors come in and out as we make our way through the code completion and as we edit and fill placeholders in templates. At the same time, chunks of data flow through the network between the workspace and frontends. Distributed transactions are rolling out underneath. It really is magic.
Summary
Code completion is just a tiny piece of Fleet’s functionality. Nowadays, it’s hard to imagine the world without this feature. Although some people find it useful to write programs in Google Docs – especially those who believe, for some reason, that it’s a reliable way to test coding abilities – the rest of us are happy to rely on code completion.
In most cases, Fleet is not smart enough to know what to suggest to its user. It uses other tools, such as IntelliJ Backend or LSP servers. Instead, Fleet focuses on delivering the best experience when editing source code in different languages. It uses all its state management machinery to help users work more efficiently.
This is not the last blog post in this series. In the next part, we’ll look at the UI framework used in Fleet. We call it Noria. Stay tuned!
Subscribe to Fleet Blog updates
