Build Chains: TeamCity’s Blend of Pipelines. Part 2 – Running Builds in Parallel

Anton Arhipov

In the previous blog post, we learned about snapshot dependencies and how they can be applied to create build chains in TeamCity. In this blog post, we describe how snapshot dependencies enable parallel builds.

More snapshot dependencies

Previously, we started creating the build chain for the demo application. We created two build configurations: one builds the application and the other builds the Docker image. What about tests?

Suppose there are a lot of tests and running those tests sequentially takes a lot of time. It would be nice to execute the groups of tests in parallel. We can create two build configurations, Test1 and Test2, which will execute the different groups of the tests. Both Test1 and Test2 have a snapshot dependency on TodoImage build configuration.

Since Test1 and Test2 do not depend on each other, TeamCity can execute these build configurations in parallel if there are build agents available.

Now the new aspect of snapshot dependencies is revealed. Using the snapshot dependencies, we created a build chain which is actually a DAG – Directed Acyclic Graph. The independent branches of the graph can be processed in parallel given that there are enough processing resources, i.e. build agents.

This leads us to the next question: how can we trigger such a build chain? Previously, we added the trigger to TodoImage as it was the last build configuration in the chain. Now there are two build configurations. Should we add two triggers – one to Test1 and the other to Test2? While it is certainly an option, there is a more idiomatic way to do that – with the Composite type of build configuration.

Composite build configuration

The purpose of the composite build configuration is to aggregate results from several other builds combined by snapshot dependencies, and present them in a single place. One very interesting property of this kind of build configuration is that it does not occupy a build agent during the execution.

In our example, we can create a composite build configuration that depends on Test1 and Test2 via a snapshot dependency. The new configuration will be the last one in the build chain. Now it’s possible to add a VCS trigger to that build configuration. As a result, there will be just one VCS trigger for the whole build chain.

Notice on the screenshot above that the two build configurations, Test1 and Test2, are running in parallel. The TestReport build configuration is also running, but it doesn’t occupy the build agent and will be marked as finished as soon as all the builds complete.

The nice feature of the composite build configuration is that it also aggregates the test results from the dependencies. In our example, if we navigate to the Tests tab of the TestReport build configuration, we will observe the list of tests that were executed in all the previous build configurations that belong to the same build chain.

Summary

At first sight, snapshot dependency looks like a simple concept. However, it enables a lot of features in TeamCity. In the previous blog post, we saw how we can re-use the results of the builds to save build time and resources. In this blog post, we have learned that snapshot dependencies also enable better use of resources when builds are executed in parallel. Next, we will learn how to orchestrate the deployment process with the TeamCity build chains.

If you are interested in trying out the demo project on your local TeamCity instance, we have uploaded the configuration to a GitHub repository. The .teamcity/ directory contains Kotlin DSL settings for the build chain that we have described in this blog post. To import these settings, create a project from repository URL. TeamCity will detect .teamcity/ directory in the repository and will suggest creating the project using these settings.

Subscribe to TeamCity Blog updates

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!