Building a Testing Framework for Distributed IDEs

Evgeniy Stepanov Maarten Balliauw

Testing is a crucial aspect of software development, as it helps ensure the quality and reliability of the final product. However, when it comes to testing an Integrated Development Environment (IDE) that operates on multiple machines and platforms, the task becomes even more challenging. How do you run tests across different machines, keep them in sync, and preserve the test authoring experience?

In this post, we’ll look at why and how we implemented distributed testing for our Remote Development and Code With Me tools.

Why we need distributed testing for our IDEs

Most of the IDEs we develop at JetBrains are based on the IntelliJ Platform. To ensure their quality and functionality, we conduct thorough testing. Historically, we have divided our tests into two categories:

1. Unit tests, based on the junit or testng framework, are the most common way to test logic. In these tests, we start the IDE in headless mode – without a UI and with some overridden components. The IDE runs inside the unit test process, allowing us to easily write test code and debug it afterwards. 
2. UI Scripting Tests are designed to test the user interface of the IDE. These tests start a separate IDE process with a UI and a real component container and communicate with it using a command system. There are relatively few of these tests, as they take a significant amount of time to run and are more difficult to write. To create these tests, a special command must be implemented within the application and can only be called by name.

These testing methods have been sufficient for developing and maintaining the quality of the IntelliJ Platform for many years. However, with the introduction of new products such as Code With Me and Remote Development, new testing challenges have emerged. These products allow you to connect to another IDE running on another computer.

• Remote Development allows you to run an IDE on a remote machine, usually in headless mode, and connect to it using a thin client on a local machine.
• Code With Me assumes that you already have an IDE with an open project and enables you to connect to it from another computer to collaborate on the project. It supports multiple participants.

In both scenarios there are two or more IDE processes running simultaneously. None of the existing test frameworks support this scenario, so we had to develop a new one!

Building a new testing framework for distributed IDEs

To effectively test our Remote Development tools and Code With Me, we needed to create a testing framework that would give us the ability to:

1. Run multiple IDE processes in various environments, possibly even on different machines. A typical test case is running a server-side IDE on Linux and having the client connect to it from Windows.
2. Run IDEs in both UI and non-UI modes.
3. Write test code easily.
4. Debug test code efficiently.

In our scenario, it is not possible to run IDEs within the unit test process. However, we still want to be able to write and debug tests as easily as possible.

We already have a special RD protocol in place, which was initially developed for Rider to connect the ReSharper backend process to the IntelliJ Frontend process. Later it was also used in Code With Me and then in Remote Development to connect the IntelliJ backend to the IntelliJ Thin Client. Why not use the same protocol here as well? It is a proven solution that can be easily integrated into our test framework.

This is our principal architecture diagram: 

The basic unit test process, either junit or testng, is initiated by the IDE. This unit test process will spawn other IDE processes – one for the server, and others for clients. For Remote Development, there will be a single client. However, for Code With Me there could be multiple clients. All of them are connected to each other and to the unit test process via the RD protocol.

The new test framework has been named the “Distributed Test Framework” as it is designed to run tests that are distributed across multiple processes or machines.

What does a simple test in this distributed test framework look like? Let’s take a look at an example:

class Test : DistributedTestBase() {

  @Test
  fun testEditor() {
    startServerAndClient(...) { server, client ->

      server.perform {
        // open an editor        
      }

      client.perform {
        // assert editor is opened
      }
    }
  }
}

The test consists of multiple perform calls that correspond to the different IDEs being tested. Each call uses the regular IDE API, but the code is executed in other processes. How does this work?

1. The startServerAndClient method runs two IDE processes by building a classpath. It creates the default IDE classpath from all core modules and then adds specific test jars to it. 
2. The testing process subsequently establishes a connection to them using the RD protocol and waits for the IDEs to be fully loaded.
3. Next, it dispatches the test class and specific test method to be executed to all of the connected IDEs.
4. The IDEs create a test class (it is already present in the class loader) and execute the specified test method.
5. Then a test executes the specified test method on its own.

The key is the perform method. It is invoked multiple times – once by the unit test process and again by the IDE processes. When it is called by an IDE process, it stores the action in a queue. In the unit test process, it sends a signal to the corresponding IDE, which then proceeds to execute the requested action from the queue. The overall structure of the method can be summarized as follows:

protected fun AgentConnection.perform(action: AgentContext.() -> Unit) {
  if (agentDescription != null) {
    // That means now we are just playing the test inside agent, so we should not perform this action
    //  but just put in into the agent queue
    if (this.agentId == agentDescription.info.agentId) {
      agentDescription.queue.add(AgentAction(action))
    }
    else {
      // Just ignore, this command is intended to other agent
    }
    return
  }

  // We are running test in a test process so send command signal to the corresponding agent
  val connection = this as PhysicalAgentConnection
  connection.scheduler.executeSync { connection.session.runNextAction.sync() }
}

This process can be illustrated by the following diagram:

This framework allows you to write code for multi-process testing as if it were for a single process. But what about debugging? Modern IDEs are equipped with advanced features and can attach a debugger to the unit test process. However, it may not be possible to set breakpoints within the perform methods as they are executed in separate processes, and the IntelliJ debugger does not have the capability to resolve this issue.

Even powerful IDEs like IntelliJ IDEA do not have the capability to start debuggers for processes started by the IDE. However, as IDE developers, we can solve this issue!

We have developed an HTTP handler that listens for requests and attaches a debugger to the requested process. We have incorporated it into devkit, a specialized section of code that enables IntelliJ developers to work more efficiently.

In the test framework, we execute the following code immediately after the IDE process is started. This ensures that the debugger will be connected to the newly started process, allowing for debugging capabilities while the test is running:

 private fun attachIDEToProcess(debuggerPort: Int): Boolean {
    val url = URL("http://localhost:63342/debug/attachToTestProcess")
    val connection = url.openConnection() as HttpURLConnection
    connection.requestMethod = "POST"
    connection.useCaches = false
    connection.doOutput = true

    DataOutputStream(connection.outputStream).use { it.writeBytes(debuggerPort.toString())  }
    try {
      return connection.responseCode == 200
    }
    catch (ex: Throwable) {
      return false
    }
  }

The debugger will only be able to connect to the specified port if it has already been opened by the JVM. Therefore, we start our IDEs with the following command line argument to ensure the port is open:

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=$port

Once the unit test process and IDEs have been initialized, we can set breakpoints and debug multiple processes as if they were a single process! This requires no additional effort from the developers – it works seamlessly out of the box:

Conclusion

By introducing this innovative test framework, we have revolutionized the way tests are written and debugged for our distributed applications. The framework is designed to make the process as seamless and effortless as possible for developers.

Sometimes, investing a little time into improving tooling can have a significant impact on the productivity of the entire team. In this case, it took only 2-3 days of work to create this framework, and now our team is able to save hours of time, be more confident when writing these kinds of tests, and focus on delivering high-quality code.

Want to join our Code With Me and Remote Development team and build a distributed IDE with us? We’re hiring!

Subscribe to JetBrains Blog updates

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!