Configuration as Code, Part 4: Extending the TeamCity DSL

Anton Arhipov

1. Getting started with Kotlin DSL
2. Working with configuration scripts
3. Creating build configurations dynamically
4. Extending Kotlin DSL
5. Using libraries
6. Testing configuration scripts

TeamCity allows us to create build configurations that are dependent on one another, with the dependency being either snapshots or artifacts. The configuration for defining dependencies is done at the build configuration level. For instance, assuming that we have a build type Publish, that has a snapshot and artifact dependencies on Package, we would define this in the build type Publish in the following way:

object Package : BuildType({
   name = "Package"

   artifactRules = “application.zip”

   steps {
       // define the steps needed to produce the application.zip
   }
})

object Publish: BuildType({
   name="Publish"

   steps {
       // define the steps needed to publish the artifacts
   }

   dependencies {
       snapshot(Package){}
       artifacts(Package) {
           artifactRules = "application.zip"
       }
   }
})

and in turn, if Package had dependencies on previous build configurations, we’d define these in the dependencies segment of its build configuration.

TeamCity then allows us to visually see this using the Build Chains tab in the user interface:

The canonical approach to defining build chains in TeamCity is when we declare the individual dependencies in the build configuration. The approach is simple but as the number of build configurations in the build chain grows it becomes harder to maintain the configurations.

Imagine there’s a large number of build configurations in the chain, and we want to add one more somewhere in the middle of the workflow. For this to work, we have to configure the correct dependencies in the new build configuration. But we also need to update the dependencies in the existing build configurations to point at the new one. This approach does not seem to scale well.

But we can work around this problem by introducing our own abstractions in TeamCity’s DSL.

Defining the pipeline in code

What if we had a way to describe the pipeline on top of the build configurations that we define separately in the project? The pipeline abstraction is something we need to create ourselves. The goal of this abstraction is to allow us to omit specifying the snapshot dependencies in the build configurations that we want to combine into a build chain.

Assume that we have a few build configurations: Compile, Test, Package, and Publish. Test needs a snapshot dependency on Compile, Package depends on Test, Publish depends on Package, and so on. So these build configurations compose a build chain.

Let’s define, how the new abstraction would look. We think of the build chain described above as of a “sequence of builds”. So why not to describe it as follows:

project {
    sequence {
        build(Compile)
        build(Test)
        build(Package)
        build(Publish)
    }
}

Almost immediately, we could think of a case where we need to be able to run some builds in parallel.

project {
    sequence {
        build(Compile)
        parallel {
            build(Test1)
            build(Test2)
        } 
        build(Package)
        build(Publish)
    }
}

In the example above, Test1 and Test2 are defined in the parallel block, both depend on Compile. Package depends on both, Test1 and Test2. This can handle simple but common kinds of build chains where a build produces an artifact, several builds test it in parallel and the final build deploys the result if all its dependencies are successful.

For our new abstraction, we need to define, what sequence, parallel, and build are. Currently, the TeamCity DSL does not provide this functionality. But that’s where Kotlin’s extensibility proves quite valuable, as we’ll now see.

Creating our own DSL definitions

Kotlin allows us to create extension functions and properties, which are the means to extend a specific type with new functionality, without having to inherit from them. When passing extension functions as arguments to other functions (i.e. higher-order functions), we get what we call in Kotlin Lambdas with Receivers, something we’ve seen already in this series when Generalising feature wrappers in the second part of this series. We will apply the same concept here to create our DSL.

class Sequence {
   val buildTypes = arrayListOf()

   fun build(buildType: BuildType) {
       buildTypes.add(buildType)
   }
}

fun Project.sequence(block: Sequence.() -> Unit){
   val sequence = Sequence().apply(block)

   var previous: BuildType? = null

   // create snapshot dependencies
   for (current in sequence.buildTypes) {
       if (previous != null) {
           current.dependencies.snapshot(previous){}
       }
       previous = current
   }

   //call buildType function on each build type
   //to include it into the current Project
   sequence.buildTypes.forEach(this::buildType)
}

The code above adds an extension function to the Project class and allow us to declare the sequence. Using the aforementioned Lambda with Receivers feature we declare that the block used as a parameter to the sequence function will provide the context of the Sequence class. Hence, we will be able to call the build function directly within that block:

project {
    sequence {
         build(BuildA)
         build(BuildB) // BuildB has a snapshot dependency on BuildA
    }
} 

Adding parallel blocks

To support the parallel block we need to extend our abstraction a little bit. There will be a serial stage that consists of a single build type and a parallel stage that may include many build types.

interface Stage

class Single(val buildType: BuildType) : Stage

class Parallel : Stage {
   val buildTypes = arrayListOf()

   fun build(buildType: BuildType) {
       buildTypes.add(buildType)
   }
}

class Sequence {
   val stages = arrayListOf()

   fun build(buildType: BuildType) {
       stages.add(Single(buildType))
   }

   fun parallel(block: Parallel.() -> Unit) {
       val parallel = Parallel().apply(block)
       stages.add(parallel)
   }
}

To support the parallel blocks we will need to write slightly more code. Every build type defined in the parallel block will have a dependency on the build type which was declared before the parallel block. And the build type declared after the parallel block will depend on all the build types declared in the block. We’ll make the assumption that a parallel block cannot follow a parallel block, though it’s not a big problem to support this feature.

fun Project.sequence(block: Sequence.() -> Unit) {
   val sequence = Sequence().apply(block)

   var previous: Stage? = null

   for (current in sequence.stages) {
       if (previous != null) {
           createSnapshotDependency(current, previous)
       }
       previous = current
   }

   sequence.stages.forEach {
       if (it is Single) {
           buildType(it.buildType)
       }
       if (it is Parallel) {
           it.buildTypes.forEach(this::buildType)
       }
   }
}

fun createSnapshotDependency(stage: Stage, dependency: Stage){
   if (dependency is Single) {
       stageDependsOnSingle(stage, dependency)
   }
   if (dependency is Parallel) {
       stageDependsOnParallel(stage, dependency)
   }
}

fun stageDependsOnSingle(stage: Stage, dependency: Single) {
   if (stage is Single) {
       singleDependsOnSingle(stage, dependency)
   }
   if (stage is Parallel) {
       parallelDependsOnSingle(stage, dependency)
   }
}

fun stageDependsOnParallel(stage: Stage, dependency: Parallel) {
   if (stage is Single) {
       singleDependsOnParallel(stage, dependency)
   }
   if (stage is Parallel) {
       throw IllegalStateException("Parallel cannot snapshot-depend on parallel")
   }
}

fun parallelDependsOnSingle(stage: Parallel, dependency: Single) {
   stage.buildTypes.forEach { buildType ->
       singleDependsOnSingle(Single(buildType), dependency)
   }
}

fun singleDependsOnParallel(stage: Single, dependency: Parallel) {
   dependency.buildTypes.forEach { buildType ->
       singleDependsOnSingle(stage, Single(buildType))
   }
}

fun singleDependsOnSingle(stage: Single, dependency: Single) {
   stage.buildType.dependencies.snapshot(dependency.buildType) {}
}

The DSL now supports parallel blocks in the sequence:

parallel {
  sequence {
    build(Compile)
    parallel {
       build(Test1)
       build(Test2)
    }
    build(Package)
    build(Publish)
  }
}

We could extend the DSL even further to support nesting of the blocks by allowing defining the sequence inside the parallel blocks.

project {
   sequence {
       build(Compile) 
       parallel {
           build(Test1) 
           sequence {
              build(Test2) 
              build(Test3)
           } 
       }
       build(Package) 
       build(Publish) 
   }
}

Nesting the blocks allows us to create build chains of almost any complexity. However, our example only covers snapshot dependencies. We haven’t covered artifact dependencies here yet and these would be nice to see in the sequence definition as well.

Adding artifact dependencies

For passing an artifact dependency from Compile to Test, simply specify that Compile produces the artifact and Test requires the same artifact.

sequence {
   build(Compile) {
      produces("application.jar")
   }
   build(Test) {
      requires(Compile, "application.jar")
   }
}

produces and requires are the new extension functions for the BuildType:

fun BuildType.produces(artifacts: String) {
   artifactRules = artifacts
}

fun BuildType.requires(bt: BuildType, artifacts: String) {
   dependencies.artifacts(bt) {
       artifactRules = artifacts
   }
}

We also need to provide a way to execute these new functions in the context of BuildType. For this, we can override the build() function of the Sequence and Parallel classes to accept the corresponding block by using Lambda with Receivers declaration:

fun Sequence.build(bt: BuildType, block: BuildType.() -> Unit = {}){
   bt.apply(block)
   stages.add(Single(bt))
}

fun Parallel.build(bt: BuildType, block: BuildType.() -> Unit = {}){
   bt.apply(block)
   stages.add(Single(bt))
}

As a result, we can define a more complex sequence with our brand new DSL:

sequence {
   build(Compile) {
       produces("application.jar")
   }
   parallel {
       build(Test1) {
           requires(Compile, "application.jar")
           produces("test.reports.zip")
       }
       sequence {
           build(Test2) {
               requires(Compile, "application.jar")
               produces("test.reports.zip")
           }
           build(Test3) {
               requires(Compile, "application.jar")
               produces("test.reports.zip")
           }
       }
   }
   build(Package) {
       requires(Compile, "application.jar")
       produces("application.zip")
   }
   build(Publish) {
       requires(Package, "application.zip")
   }
}

Summary

It’s important to understand that this is just one of many ways in which we can define pipelines. We’ve used the terms sequence, parallel and build. We could just as well have used the term buildchain to align it better with the UI. We also added the convenience methods to the BuildType to work with the artifacts.

The ability to easily extend the TeamCity DSL with our own constructs, provides us with flexibility. We can create custom abstractions on top of the existing DSL to better reflect how we reason about our build workflow.

In the next post in our Configuration as code series, we’ll see how to extract our DSL extensions into a library for further re-use.

Subscribe to TeamCity Blog updates

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!