What Every Rust Developer Should Know About Macro Support in IDEs
We use a lot of tools for software development. Compilers, linkers, package managers, code linters, and, of course, IDEs are essential parts of our work and life. There are areas where single-tool efforts are not enough to provide the best user experience. In Rust, macro support is definitely something we can’t wholly tackle without broad community understanding and collaborative effort.
We, the IntelliJ Rust plugin team, are now partially enabling support for procedural macros, specifically enabling function-like and derive procedural macro expansion by default while hiding support for attribute procedural macros behind the org.rust.macros.proc.attr experimental feature flag. While we mostly refer to the IntelliJ Rust plugin here, the same things apply to your favorite editor powered by rust-analyzer. In fact, we are very similar regarding macro support. Even more importantly, we face the same problems. This more technical blog post by Lukas Wirth explores the same area.
Let’s discuss several fundamental ideas regarding macros and their support in IDEs, including main ideas and approaches, good and bad parts, implementation details, and problems.
Learning about Rust macros and taking IDEs into account
Most Rust developers aren’t concerned about implementing macros, but we use them a lot. Macros simplify common operations (println! and vec!), reduce boilerplate code (like serde), provide additional features for our programs by generating a lot of code (clap or actix-web), or allow us to embed DSL code in our Rust programs (notably yew).
The Rust Programming Language book provides a short but accessible overview of macros, their flavors, and their kinds. It also gives you a glimpse of how they can be implemented. Unfortunately, The Book doesn’t go into detail about how macros actually work, although understanding this is crucial to get an idea of what’s going on in an IDE whenever we use them.
Also unfortunately, most available online educational resources about macros ignore the IDE experience. Understanding macros based on a compiler-only experience is a bit misleading. The goal of a compiler is simply to check your code and either report an error or build an executable/library. For the compiler, there is no difference if there is an error in a macro implementation or in the code where we call that macro. Macro authors usually share the same attitude towards code they’ve gotten from a user: macros either do their job or report an error regarding the input they received. If we add an IDE into this equation, things change a lot. Most of the time, an IDE deals with incorrect code. Although we expect IDEs to report errors, their primary goal is not to complain about the inability to do this and that! IDEs drive users to the correct code by staying alive in the presence of errors and suggesting fixes. The compiler is of almost no help to an IDE here, but we do expect some help from the macro implementor – more on this later.
Turning code into code with macro expansion
Macros take code as an argument and are able to add new code, replace a given code with anything else, or modify this code in any way imaginable. This is important! Whenever you provide some code as an argument to a macro, chances are you don’t know what is actually going to be compiled in the end because the code can be heavily modified. This input code doesn’t have to be valid Rust (although there are some technical restrictions), but the resulting code must be valid Rust.
The process of executing a macro is called macro expansion. Macros are expanded when our code is compiled. More interestingly, they should also be expanded when we write our code in an IDE. Why? Because an IDE should be aware of expanded code in order to provide us with reasonable navigation and code completion. For example, if a macro generates some new function, we should see it in the completion suggestions, although there may be no place in the code to navigate to. A user writes and sees code before macro expansion happens, but an IDE is expected to provide an experience as if it has already happened.
Look at the diagram below. Suppose a user expects some IDE services at points A and B in the source code. Point A is inside a macro call, and point B follows it. In order to provide these services, an IDE has to expand the macro first. Then, it analyzes the expanded code, comes up with the necessary information, and delivers it to the user.
If a macro fails to expand properly, an IDE is in trouble and its ability to deliver helpful information to a user is severely limited.
The two flavors of macros
Macros come in two flavors: declarative and procedural. These flavors mainly refer to the ways in which macros are implemented, not how they are used. They also differ in the ways the compiler and IDEs work with them. For declarative macros, the IntelliJ Rust plugin uses its own built-in expansion mechanism. For procedural macros, the story is much more complicated. In short, Rust libraries that provide procedural macros are compiled into dynamic libraries for a corresponding operating system. These libraries are then called by the compiler or IDE at the moment of macro expansion. While the compiler uses its own mechanism for calling these libraries, the macro invocations from IDEs are facilitated by proc_macro_srv, a server for procedural macro expansion. The following diagram outlines these macro expansion processes for both declarative and procedural macro calls.
The proc_macro_srv macro expander we use is a component developed by the Rust Analyzer team. Their implementation is based on code originally written by a student during an internship at JetBrains. The procedural macro expander has become so ubiquitous that it is now included in the compiler distribution itself and is shipped by rustup.
If you feel adventurous and are interested in the technical details behind procedural macros, you can read the corresponding series in our blog (part I, part II). You can also dive into this epic story by Amos, where he explains how proc_macro_srv made its way into the compiler distribution.
IDEs prefer declarative macros because the machinery to expand them is significantly simpler and usually more stable. Procedural macros demand much more care, and the machinery is more fragile.
Three kinds of procedural macros and how we deal with them
There are three kinds of procedural macros: derive, function-like, and attribute. These kinds of macros work mostly the same way in terms of macro expansion, but they are used for different things and put different limitations on the code used as an argument. We’ll use code from this repository to showcase the usage of different kinds of procedural macros. Feel free to open this repository in your IDE and explore IDE support on your own.
Derive macros
Derive macros expect valid struct, enum, or union declarations as input. They can’t modify their input in any way, and generate new code (mostly impl-blocks) based on their input. Derive macros are relatively straightforward in terms of IDE support. All of the newly generated code is included in code analysis, so the IDE takes it into account.
Let’s look at an example. Suppose we have the following trait:
trait Name {
fn name() -> String;
}
We also have a macro that derives the name function for a struct or enum it is applied to:
#[proc_macro_derive(NameFn)]
pub fn derive_name_fn(items: TokenStream) -> TokenStream {
fn ident_name(item: TokenTree) -> String {
match item {
TokenTree::Ident(i) => i.to_string(),
_ => panic!("Not an ident")
}
}
let item_name = ident_name(items.into_iter().nth(1).unwrap());
format!("impl Name for {} {{
fn name() -> String {{
\"{}\".to_string()
}}
}}", item_name, item_name).parse().unwrap()
}
In this macro, we take an identifier of an item it is applied to and emit the corresponding impl block. Note that the implementation is kept fragile for simplicity. For example, we assume that the item’s identifier comes right after the introducing keyword for the item (struct, enum, or union), so we skip the keyword and then take the next token.
With these definitions available, we can use them in our code as follows:
#[derive(NameFn)]
struct Info;
#[derive(NameFn)]
enum Bool3 { True, False, Unknown }
As you can see in the screenshots below, the IntelliJ Rust plugin now has all of the information it needs to assist us with using the name function:
Function-like macros
Function-like procedural macros are invoked using the ! operator. They expect any sequence of code fragments (called tokens) that can be grouped in brackets, braces, or parentheses, and they output valid Rust code built from the input. As a result of macro expansion, the call site is replaced with the output. Once again, the user sees a macro called in code, but the IDE should make it feel like the expanded code is in place.
As an example, we’ll implement a function-like macro that can be used as follows:
declare_variables! {
a = 'a',
b = 2,
c = a,
d, // will be defaulted to 0
e = "e",
}
This macro provides a short syntax for declaring variables. Thanks to the “Show recursive macro expansion” context action, we can see the result of expansion:
Every item in the shortened declaration list gets expanded into a full Rust variable declaration. There are no declarations of these variables in the source code, but the IntelliJ Rust plugin is smart enough to provide us with code completion and navigation, as seen in the screenshots below:
This IDE behavior is not a given. If we look at the macro implementation, we’ll see that original input tokens make their way to the output let declarations:
let variable = decl.next().unwrap();
// ...
tokens.extend([// construct a new let-declaration
Ident::new("let", Span::mixed_site()).into(),
variable, // comes from input
Punct::new('=', Spacing::Alone).into(),
value,
Punct::new(';', Spacing::Alone).into()
]);
If we attempt to construct let declarations from a String literal as in the derive macro above, there would be no such connection between a variable declaration site and its usages. As we mentioned previously, an IDE’s abilities depend greatly on the macro implementation.
Attribute macros
Attribute macros are attached to existing code items which must be valid Rust code. Attribute macros can play with their input in any way they want, and their output fully replaces the input. Consider the following as a counterintuitive example of an attribute macro, but keep in mind that attribute macro support in the IntelliJ Rust plugin is not yet enabled by default:
#[proc_macro_attribute]
pub fn clear(_attr: TokenStream, _item: TokenStream) -> TokenStream {
TokenStream::new()
}
This macro removes a code item it is applied to. Considering this, what do you think about the following IDE suggestions?
If we apply the first one, we would expect no problems – the report_existence function is eradicated during compilation, along with the call to a no-more-in-existence function. On the other hand, applying the second suggestion would lead to a compilation error:
error[E0425]: cannot find function `report_existence` in this scope --> demo/src/main.rs:20:5 | 20 | report_existence(); | ^^^^^^^^^^^^^^^^ not found in this scope
If we have an external linter enabled, we would know that in advance, but it would be better not to suggest this in the first place. In fact, we can avoid these issues with the org.rust.macros.proc.attr experimental feature flag. If we enable it, there will be no such suggestion:
Does it make sense to suggest anything at this point? In these circumstances, we provide a generic list of suggestions as if there was no attribute at all. We believe that these suggestions can be useful in dealing with erroneous situations around macro invocations.
We still have some work to do regarding our support for attribute procedural macros. For example, we have to make sure that enabling it doesn’t break the user experience. The expansion of attribute macros may create tension between a user who types some code and an IDE pretending something totally different (the result of macro expansion) is in place. This is an important difference from function-like macros. If users type some arbitrary tokens inside a function-like macro call site, their expectations regarding IDE support are significantly lower compared to attribute macro inputs, which have to be valid Rust. Ultimately, it’s user expectations that drive users towards public complaints and issues in issue trackers! Understanding the reasons behind awkward IDE behavior in the presence of procedural macro invocations should help.
Now, let’s send a message to our fellow macro implementers.
What every Rust macro implementor should take into account
We don’t aim to provide a complete guide for macro implementers. Instead, we’d like to bring your attention to several issues that could help IDEs if addressed adequately by those who write their own macros.
Try to write a declarative macro if possible. IDE machinery is much easier and more efficient when it comes to declarative macro expansion.
If you still want to write a procedural macro, avoid stateful computations and I/O. It’s never guaranteed when and how many times macro expansion is going to be invoked. Connecting to a database every time a user waits for completion suggestions seems unnecessary.
Procedural macros process tokens that come in TokenStreams. Every input token bears its location in the source code known as a span. IDEs can use these spans for syntax highlighting, code navigation, and error reporting. If you want an IDE to be able to navigate to something generated from those tokens, reuse those spans. The more input reused in output, the better. This allows the IDE to provide better user assistance.
What if macro input is malformed? The Rust Reference clearly states that:
Procedural macros have two ways of reporting errors. The first is to panic. The second is to emit a compile_error macro invocation.
Note that the compiler-centric approach used here is definitely not IDE-friendly, given that IDE has to expand macros on-the-fly as users type their code. Why do IDEs have to be so quick in expanding macros? Because expanding macros gives much more information about the context that can be used for code completion suggestions and navigation. An IDE (and the user) needs that information.
The critical point is that an IDE expands macros not to compile or run code, but to get information. If a macro invocation results in error, this information is lost. But is there an alternative? If there is an error in the input, there should be an error in the macro invocation! It also seems like a mistake to demand macro authors rewrite their parsers so that they accept malformed syntax, recover from errors, and try to come up with suggestions on how to fix them. Writing such parsers can be quite challenging. Neither macro implementers nor the proc_macro ecosystem seems ready for that. We also are not sure that they should, even if they could.
We believe that there is an easier way. Both IntelliJ Rust and Rust Analyzer invoke macro expansion for computing completion suggestions. They mark the caret position (an expected insertion point) with a dummy identifier to be able to compute code completion suggestions around it. We also make it clear for macro implementers that the invocation is done for completion purposes. Such invocations can be thought of as a side channel between the IDE and macro implementers that can be used to deliver valuable information (about the macro inputs, for example) to expose potentially available names or expected grammar.
Look at this pull request to the yew library aimed to improve the user experience with the completion of component names inside an html! macro invocation. While being more of a proof of concept, it should encourage macro implementers to care more about the user experience in IDEs without much hassle.
Let’s make writing Rust code easier together! And let’s continue closing those annoying issues and implementing new exciting features.
Your IntelliJ Rust plugin team
Subscribe to Rust Blog updates
