Update on running ReSharper out of process

Matt Ellis

Last time we talked about ReSharper out of process, we took a (very long!) look at the magnitude of the task. It’s a massively complex job, re-architecturing an application with 14 years worth of assumptions of COM based APIs into an asynchronous cross process model.

That post details the complexity of moving the project model out of process – migrating from calling Visual Studio’s COM APIs on the UI thread, to using the MSBuild APIs in a separate process. This is now done, and has been the default way we manage the list of projects, files and references since ReSharper 2019.2.

As we’ve just been talking about our roadmap for ReSharper 2020.1, it’s time to ask the obvious question: where are we now?

Current internal builds are running a large part of ReSharper out of process, although it’s definitely not fully functional. Joining the project model, ReSharper’s code model is now running out of process too. This includes all of the language parsing, analysis and inspections, caching, syntactic and semantic highlighting as well as the Alt+Enter menu and quick fixes and context actions. This is a significant milestone, although it’s not quite complete – we’re still working on document and text control handling and integration.

While this step is significant, there’s still a lot to do before this is usable and ready for release. Most of ReSharper’s features work on top of the code model, and these features have not yet been updated to work when running out of process. For example, an action that is invoked from a menu item needs to run out of process, but might still need data from Visual Studio, such as which item is selected in Solution Explorer or open in the editor. We can pass all the necessary data to the ReSharper process, but need to balance this carefully, as fetching the data requires time on the UI thread.

Similarly, refactorings require UI running in the Visual Studio process, based on data from the backend ReSharper process. We’ve made a lot of changes already for Rider that will apply here, and we’re hoping this makes the task easier. But we still need all of these features to be migrated before we can say this is done.

We’ll be continuing this work during 2020. We won’t have anything to try in 2020.1, but we do plan to release experimental builds as the year progresses.

P.S. …

If you’re interested in some of the technical challenges of running ReSharper out of process, read on for a taste of some of the things we’ve been working on since our last post. Feel free to skip this section – this is only for the curious!

One of the biggest challenges is that this massive architectural refactoring has to happen in our main development branch. This is a long term refactoring. It’s not something we can stop the world and fix – we need to keep up to date with new versions of C# and Visual Studio, so we need to keep ReSharper moving forward, and make the changes for out of process.

In other words, the architecture has to work both in-process and out of process, all from the same code base. We can’t have two implementations of everything, as that will double our testing load, and will inevitably lead to bugs as the implementations diverge. So we’ve been refactoring internal components to run either in-process or out of process, and we can even toggle between in and out of process dynamically at run time!

But this leads to complexity too. When our component container creates new object instances, and passes in dependencies, it needs to know if those dependencies are in or out of process. And it can be different each time it’s run!

The process of loading a solution is also significantly affected by moving to an out of process model. There are specific milestones during solution load where we need to perform tasks. For example, building caches requires the list of files in a project, so we need to load the project model first. Various other steps require the caches, and so on.

Even though we’re moving ReSharper to run out of process, this doesn’t mean that all of ReSharper will be running in a separate process – there are still parts that need to live in the Visual Studio process, and yes, you’ve guessed it, some of them are solution load tasks (e.g. we still get some project model details from Visual Studio’s COM APIs for projects that don’t support MSBuild).

All of this means that we have to synchronise solution load tasks across two processes. Fortunately, these problems have been solved, and we can now construct ReSharper in or out of process, load solutions, load the project model, and build and use the code model to run analysis and quick fixes.

Before we can move on, we need to finish off the work with documents and text controls. Obviously, the Visual Studio editor integration needs to remain in the Visual Studio process, but we want everything else to be in the external ReSharper process. And of course, we need it to work correctly both in and out of process. But once this is done, we get to move up a layer, and start work on the action system, and more UI-facing features!

So while we don’t have anything visible in ReSharper 2020.1, a lot of updates are happening towards making ReSharper out of process a reality.

Comments below can no longer be edited.

15 Responses to Update on running ReSharper out of process

  1. Ryan says:

    February 24, 2020

    Thank you for the update.

  2. Adam says:

    February 25, 2020

    I really appreciate that you are not giving up and actively working on complete “out of process” implementation. Fingers crossed and looking forward to out of process ready EAPs.

  3. Alexander Chrostowski says:

    February 25, 2020

    Thank you! Looking forward to trying any experimental out of process ReSharper builds.

  4. Tomaz Burowski says:

    February 25, 2020

    nice

  5. Paul says:

    February 25, 2020

    I think you’re all brave for taking on this challenge. I can imagine you’ve had to pull everything apart and put it back together to get it working. I also enjoyed reading the challenges you’re facing and how you’re getting around them, be great to hear more. Really looking forward to seeing the results of your work.

  6. ChrisSolutions says:

    February 25, 2020

    It’s amazing that Visual Studio still supports COM after all this time. I thought they were going to migrate Visual Studio to purely managed code. I guess they didn’t and thus the power of the C++ lobby at Microsoft and why they are still running Visual Studio as 32-bit.

    • Mike Diack says:

      February 27, 2020

      ChrisSolutions, don’t forget that under the bonnet (“hood”) much of managed code is implemented via COM…..

  7. Jeff Chen says:

    February 26, 2020

    Thank you for the update! I am enjoy to read it.

  8. Mike Diack says:

    February 27, 2020

    Can’t help thinking that you should have bitten the bullet and pulled all your efforts to fixing this ASAP on the main development branch at the cost of temporarily not tracking the latest Microsoft innovations, rather than chasing the bleeding edge language features – would have got the work done quicker, rather than the complexity of multiple runtime versions of things, and ongoing technical debt.

    How big a proportion of your users are actually demanding i.e. using the absolute latest c#, C++ 20 features? I bet it’s much less than you think….

    • Matt Ellis says:

      February 27, 2020

      Hi Mike. Thanks for the comments. We do have opt-in stats that we use to help prioritise. We need to maintain a balance between working on out of process and supporting new language features.

      We can’t necessarily speed up out of process development by adding more people. The changes to the architecture are layered, and sequential – we can’t easily start the next layer up until the current layer is done.

      Keeping up to date with C# isn’t only about chasing the latest and greatest features, it’s just as much about maintaining existing features. New syntax breaks our existing parsers, analysis and potentially quick fixes and refactorings. We also need to support the latest versions of Visual Studio and various SDK versions. And of course ReSharper is the engine that powers Rider, so we need to update ReSharper so that Rider can keep up to date too.

      Out of process is very much in development, and we’ll have experimental builds for you to try as soon as we can.

  9. Ritch Melton says:

    March 3, 2020

    Are you tracking any sort of rough performance improvements at this point? I know one of our solutions routinely runs VS out of memory when resharper is loaded, so we don’t use it for that. But for other solutions the recent project model changes were a big win. I’d be interesting in hearing about rough-cut improvement numbers and expectations in different scenarios.

    >we’ll have experimental builds for you to try as soon as we can.

    I can’t wait to submit perf data. I’m pretty excited about this change.

    • Matt Ellis says:

      March 4, 2020

      We’re currently focused on implementation and stability, so don’t have any figures to share at the moment.

  10. Dimiter Stanev says:

    April 17, 2020

    What system are you using for IPC communication? Also how do you encode your messages? Just curious 🙂

    I also wish Visual Studio was 64-bit, then again “gc” may get worse with even more allocations at hand (which a 64-bit version would allow).

    • Matt Ellis says:

      April 30, 2020

      ReSharper out of process is using the same IPC that Rider does, which is simple sockets. We did investigate alternatives during Rider’s initial development, such as message queues, but they added overhead without really giving us anything.

      Rider’s protocol is a hierarchical, stateful, two-way observable data structure, that represents “view model” type information – the minimum amount of information required to accomplish something, such as the name, icon and id of an entry in the Alt+Enter menu. And because it’s stateful and observable, it’s very easy to get the models in sync. Theoretically, we could seamlessly restart the external process and rebuild state in the case of crashes (seamlessly from the point of view of the protocol, not necessarily the user experience or job in progress!).

      And because the data is shared between the “front end” and the “back end”, we have a schema, described in a Kotlin DSL, that we use to generate data structures. And we also use it to generate serialisation, which is very easy – read a string, read an int, read a collection. Write a string, write an int, write a collection, etc. Very simple, no reflection, very fast.

      There’s a more detailed description in this magazine article.

  11. Stephen White says:

    April 18, 2020

    Thanks for posting this Matt, I know I am a little late to respond, but I just wanted to extend my thanks to you and the rest of Jetbrains for continuing to make Resharper worthy of being called the “LEGENDARY EXTENSION”. I can already feel the performance improvements with 20.1. You guys are the best 🙂

Subscribe

Subscribe to .NET Tools updates