Integration Tests for Plugin Developers: Intro, Dependencies, and First Integration Test

Maxim Kolmakov

Following the enthusiastic response to our Plugin Testing: Performance, UI, and Functional Testing session at JetBrains Plugin Developer Conf 2024, we’re launching a series of blog posts diving deeper into plugin testing strategies. 

This first post will guide you through setting up your testing environment and creating your first integration test, with detailed step-by-step instructions.

Integration tests

You might be wondering: “Why do we need integration tests when we already have unit tests? Aren’t unit tests easier to write, maintain, and run?” This is a valid question! 

While the IntelliJ department maintains over 400,000 unit tests, we also rely on approximately 1,000 integration tests. 

Here’s why they’re essential: 

• Testing complex scenarios: Some scenarios, particularly UI interactions, cannot be effectively covered by unit tests alone. 
• Full product testing: Integration tests run against the complete product rather than isolated components. This helps identify issues that unit tests might miss, such as module interaction problems, classpath conflicts, and plugin declaration issues.
• User story validation: Integration tests typically mirror real user scenarios, ensuring your plugin works reliably from start to finish. 

Despite common perceptions, integration tests aren’t as difficult to create as you might think! Let’s explore how to set them up. 

Note: While the following instructions are based on the IntelliJ Platform Gradle Plugin, the general principles also apply to other setups.

Adding dependencies

Our integration testing framework consists of two main components:

• Starter: Handles IDE configuration, test project setup, IDE startup, and output collection. 
• Driver: Provides additional functionality we’ll explore in future blog posts.

The Starter framework exclusively supports JUnit 5, as it leverages JUnit 5’s extensions and specialized listeners that aren’t available in JUnit 4.

Add the following dependencies to your build.gradle.kts:

dependencies {
    intellijPlatform {
        // Starter Framework
        testFramework(TestFrameworkType.Starter)
    }

    // JUnit 5
    testImplementation("org.junit.jupiter:junit-jupiter:5.10.2")
    // Dependency Injection for Starter configuration
    testImplementation("org.kodein.di:kodein-di-jvm:7.20.2")
}

The testFramework will resolve Starter Framework dependencies automatically, referring to the version of the currently targeted IntelliJ Platform.

If you don’t use IntelliJ Platform Gradle Plugin, you will need to add the following libraries to your Gradle file:

// Starter Framework
testImplementation("com.jetbrains.intellij.tools:ide-starter-squashed:LATEST-EAP-SNAPSHOT")
testImplementation("com.jetbrains.intellij.tools:ide-starter-junit5:LATEST-EAP-SNAPSHOT")

// Driver Framework
testImplementation("com.jetbrains.intellij.tools:ide-starter-driver:LATEST-EAP-SNAPSHOT")
testImplementation("com.jetbrains.intellij.driver:driver-client:LATEST-EAP-SNAPSHOT")
testImplementation("com.jetbrains.intellij.driver:driver-sdk:LATEST-EAP-SNAPSHOT")
testImplementation("com.jetbrains.intellij.driver:driver-model:LATEST-EAP-SNAPSHOT")

Important notes:

• Use the same version across all framework libraries to avoid compatibility issues.
• As the Driver and UI components continue to evolve, we aim to keep the Starter API stable, with occasional breaking changes.

Integrating Starter with IntelliJ Platform Gradle Plugin

To test your plugin, the Starter framework needs to know where to find your plugin distribution for installation in the IDE. This requires configuring your Gradle test task. 

Add the following configuration to your build.gradle.kts:

tasks.test {
    dependsOn("buildPlugin")
    systemProperty("path.to.build.plugin", tasks.buildPlugin.get().archiveFile.get().asFile.absolutePath)
    useJUnitPlatform()
}

This configuration does the following:

• Makes the test task depend on buildPlugin, ensuring your plugin is built before tests run.
• Sets the path.to.build.plugin system property to point to your plugin distribution file.
• Enables JUnit Platform for test execution.

Creating the first test

Now that we’ve completed the configuration, let’s write our first integration test, which will: 

• Start the IDE with our plugin installed.
• Wait for all background processes to complete.
• Perform a shutdown.

Create a new Kotlin file in src/test/kotlin with the following code:

class PluginTest {
   @Test
   fun simpleTestWithoutProject() {
       Starter.newContext(testName = "testExample", TestCase(IdeProductProvider.IC, projectInfo = NoProject).withVersion("2024.3")).apply {
           val pathToPlugin = System.getProperty("path.to.build.plugin")
           PluginConfigurator(this).installPluginFromPath(Path(pathToPlugin))
       }.runIdeWithDriver().useDriverAndCloseIde {
       }
   }
}

Let’s break down each part of the test:

1. Context creation

Starter.newContext(testName = "testExample", TestCase(IdeProductProvider.IC, projectInfo = NoProject).withVersion("2024.3"))

The Context object stores IDE runtime configuration:

• IDE type (e.g., IntelliJ Community, PhpStorm, GoLand).
• IDE version (2024.3 in this example).
• Project configuration (using NoProject for this example).
• Custom VM options, paths, and SDK settings.

The testName parameter defines the folder name for test artifacts, which is useful when running multiple IDE instances in a single test. We’re using IntelliJ IDEA Community Edition version 2024.3, and we’re starting the IDE without any project, so the welcome screen will be shown.

2. Plugin installation

.apply {
    val pathToPlugin = System.getProperty("path.to.build.plugin")
    PluginConfigurator(this).installPluginFromPath(Path(pathToPlugin))
}

This step configures plugin installation using the plugin path we defined in our Gradle configuration.

Note: PluginConfigurator can install plugins from local paths or Marketplace.

3. IDE life cycle management

.runIdeWithDriver().useDriverAndCloseIde {
}

These two methods:

• Start the IDE instance (runIdeWithDriver).
• Shut down the IDE (useDriverAndCloseIde).

The empty lambda is used for IDE interactions (useDriverAndCloseIde).

Note: When you run the test for the first time, it may take longer than expected since it needs to download the IDE. Subsequent runs will be faster, using the cached IDE version. 

Understanding the test architecture

You can now run the test and it should pass. 

Integration tests operate across two separate processes: 

• Test process:
  • Executes your test code, sending commands to the IDE. 
  • Manages the IDE life cycle. 
  • Controls test flow and assertions. 
• IDE process:
  • Listens and executes commands from the test process. 

This dual-process architecture explains several key aspects of integration testing:  

• Why debugging requires special considerations. 
• The need for a communication protocol between test and IDE processes. 
• Why a built plugin distribution is required. 
• The origin of certain test-specific exceptions. 

Note: In upcoming blog posts, we’ll explore these aspects in detail, including debugging techniques and handling common scenarios.

Opening projects in tests

While starting an IDE with an empty project is useful, sometimes we need actual projects to verify real-world scenarios. Let’s modify our test to open a project.

The framework supports several ways to specify test projects:

• GitHub projects: GitHubProject.fromGithub( branchName = "master", "JetBrains/ij-perf-report-aggregator" )
• Remote archives: RemoteArchiveProjectInfo("https://github.com/JetBrains/intellij-community/archive/master.zip")
• Local projects: LocalProjectInfo(Path("src/test/resources/test-projects/simple-project"))

Here’s our test modified to open a GitHub project:

@Test
fun simpleTest() {
   Starter.newContext(
       "testExample",
       TestCase(
           IdeProductProvider.IC,
           GitHubProject.fromGithub(branchName = "master", repoRelativeUrl = "JetBrains/ij-perf-report-aggregator")
       ).withVersion("2024.3")
   ).apply {
       val pathToPlugin = System.getProperty("path.to.build.plugin")
       PluginConfigurator(this).installPluginFromPath(Path(pathToPlugin))
   }.runIdeWithDriver().useDriverAndCloseIde {
       waitForIndicators(5.minutes)
   }
}

While simple, this test verifies a critical aspect: our plugin doesn’t interfere with IDE startup.

Note: We added a call waitForIndicators into the lambda to make sure that we wait till all indicators are gone before exiting the IDE.

How to catch exceptions from the IDE?

Our test has one critical limitation: it won’t detect exceptions or freezes occurring within the IDE process. Let’s understand why and how to fix this. 

Due to the two-process architecture:

• Exceptions in the IDE process aren’t automatically propagated to the test process.
• A bundled plugin collects exceptions from the IDE’s MessageBus.
• These exceptions are stored in the error folder within the logs.

The Starter framework uses TeamCity reporting by default, falling back to NoCIServer for other environments. However, we can customize this behavior using Kodein Dependency Injection. Here’s how to make tests fail when IDE exceptions occur:

init {
   di = DI {
       extend(di)
       bindSingleton<CIServer>(overrides = true) {
           object : CIServer by NoCIServer {
               override fun reportTestFailure(testName: String, message: String, details: String, linkToLogs: String?) {
                   fail { "$testName fails: $message. \n$details" }
               }
           }
       }
   }
}

How it works

• We create a custom implementation of CIServer that extends NoCIServer.
• We override only the reportTestFailure method to fail the test with detailed error information.
• The Starter framework collects exceptions from the IDE and passes them to our custom implementation.
• Any IDE exception now causes the test to fail with a descriptive message.

This extensibility pattern can be applied to customize other aspects of the Starter framework as needed.

Putting it all together

Here’s our complete test implementation that forms the foundation for future plugin testing: 

import com.intellij.ide.starter.ci.CIServer
import com.intellij.ide.starter.ci.NoCIServer
import com.intellij.ide.starter.di.di
import com.intellij.ide.starter.driver.engine.runIdeWithDriver
import com.intellij.ide.starter.ide.IdeProductProvider
import com.intellij.ide.starter.models.TestCase
import com.intellij.ide.starter.plugins.PluginConfigurator
import com.intellij.ide.starter.project.GitHubProject
import com.intellij.ide.starter.runner.Starter
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.fail
import org.kodein.di.DI
import org.kodein.di.bindSingleton
import kotlin.io.path.Path

class PluginTest {
   init {
       di = DI {
           extend(di)
           bindSingleton<CIServer>(overrides = true) {
               object : CIServer by NoCIServer {
                   override fun reportTestFailure(
                       testName: String,
                       message: String,
                       details: String,
                       linkToLogs: String?
                   ) {
                       fail { "$testName fails: $message. \n$details" }
                   }
               }
           }
       }
   }

   @Test
   fun simpleTest() {
       val result = Starter.newContext(
           "testExample",
           TestCase(
               IdeProductProvider.IC,
               GitHubProject.fromGithub(branchName = "master", repoRelativeUrl = "JetBrains/ij-perf-report-aggregator")
           ).withVersion("2024.3")
       ).apply {
           val pathToPlugin = System.getProperty("path.to.build.plugin")
           PluginConfigurator(this).installPluginFromPath(Path(pathToPlugin))
       }.runIdeWithDriver().useDriverAndCloseIde {
           waitForIndicators(5.minutes)
       }
   }
}

You can find the source code here.

This test provides a robust foundation for more elaborate tests by:

• Downloading and opening a real project.
• Starting the IDE with your plugin installed.
• Waiting for all background processes to complete.
• Monitoring for exceptions and freezes.
• Performing a shutdown.

What’s next?

Stay tuned for upcoming blog posts in this series where we’ll cover:

• UI testing: How to automate interface interactions.
• API testing: Working with plugin APIs effectively via JMX calls.
• GitHub Actions: Setting up continuous integration.
• Common pitfalls: Tips and tricks for stable UI tests.

Each post will build upon this foundation, helping you create comprehensive test coverage for your plugin.

Subscribe to JetBrains Platform updates

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!