Features How-To's Tips & Tricks

Configuration as Code, Part 6: Testing Configuration Scripts

In this blog post, we are going to look at how to test TeamCity configuration scripts.

  1. Getting started with Kotlin DSL
  2. Working with configuration scripts
  3. Creating build configurations dynamically
  4. Extending Kotlin DSL
  5. Using libraries
  6. Testing configuration scripts

Given that the script is implemented with Kotlin, we can simply add a dependency to a testing framework of our choice, set a few parameters and start writing tests for different aspects of our builds.

In our case, we’re going to use JUnit. For this, we need to add the JUnit dependency to the pom.xml file


We also need to define the test directory.


In this example, we have redefined the source directory as well, so it corresponds with the following directory layout.

Once we have this in place, we can write unit tests as we would in any other Kotlin or Java project, accessing the different components of our project, build types, etc.

However, before we can start writing any code we need to make a few adjustments to the script. The reason is that our code for the configuration resides in settings.kts file. The objects that we declared in the kts file are not visible in the other files. Hence, to make these objects visible, we have to extract them into a file (or multiple files) with a kt file extension.

First, instead of declaring the project definition as a block of code in the settings.kts file, we can extract it into an object:

version = "2018.2"


object SpringPetclinic : Project ({

The SpringPetclinic object then refers to the build types, VCS roots, etc.

Next, to make this new object visible to the test code, we need to move this declaration into a file with a kt extension:


settings.kts now serves as an entry point for the configuration where the project { } function is called. Everything else can be declared in the other *.kt files and referred to from the main script.

After the adjustments, we can add some tests. For instance, we could validate if all the build types start with a clean checkout:

import org.junit.Assert.assertTrue
import org.junit.Test

class StringTests {

   fun buildsHaveCleanCheckOut() {
       val project = SpringPetclinic

       project.buildTypes.forEach { bt ->
           assertTrue("BuildType '${bt.id}' doesn't use clean checkout",

Configuration checks as part of the CI pipeline

Running the tests locally is just one part of the story. Wouldn’t it be nice to run validation before the build starts?

When we make changes to the Kotlin configuration and check it into the source control, TeamCity synchronizes the changes and it will report any errors it encounters. The ability to now add tests allows us to add another extra layer of checks to make sure that our build script doesn’t contain any scripting errors and that certain things are validated such as the correct VCS checkout, as we’ve seen above, and the appropriate number of build steps are being defined, etc.

We can define a build configuration in TeamCity that will execute the tests for our Kotlin scripts prior to the actual build. Since it is a Maven project, we can apply Maven build step – we just need to specify the correct path to pom.xml, i.e. .teamcity/pom.xml.


The successful run of the new build configuration is a prerequisite for the rest of the build chain. Meaning, if there are any JUnit test failures, then the rest of the chain will not be able to start.

Comments below can no longer be edited.

8 Responses to Configuration as Code, Part 6: Testing Configuration Scripts

  1. Avatar

    Jakub Podlešák says:

    November 8, 2019

    Hi Anton,

    I find the post very useful overall. Many thanks for it. I am only wondering why there is this mixture of Kotlin based config code and UI based build step example (this .teamcity/pom.xml maven invocation). When you already have your project configured as Kotlin code, it woud be useful to also show the rest as code.

    A more generic question is related to pull request workflow with a TeamCity based gate (as described e.g. in https://www.jetbrains.com/help/teamcity/pull-requests.html). There ideally (IMHO) if you make a pull request that contains gate update (you touch anything in this .teamcity subdirectory), it would be nice if the TC related change was reflected in what is actually build as part of the pull request build process. Is it possible at all? Is there any blueprint for such a configuration as code?

    Thanks a lot in advance for any comments and thoughts!



  2. Avatar

    Jakub Podlešák says:

    November 12, 2019

    Thanks for the quick response and the links.

    I have successfully configured a pull request gate build project as code in TeamCity, but that was not the issue. The issue that i am trying to solve now is that gate build updates (changes to .teamcity stuff, that could be part of any PR) are not tested as part of the gate build (corresponding Kotlin code is taken from the master and not from the actual PR branch). I know that this is kind of a chicken egg problem, but i hoped it should be solvable somehow (a PR gate build should be based on the corresponding branch). The thing is that the gate related Kotlin code is taken from the master branch (maybe a configuration issue, but i do not know how to configure it otherwise). The gate is run, and if it succeeds, and the branch is then merged with the master, the following build could break (because the gate program changed with the previous commit, and the change was not tested). Any thoughts on that?

    • Avatar

      Anton Arhipov says:

      November 13, 2019

      Yeah, it is a chicken and egg problem, you just have to choose who’d should be the first 🙂

      In fact, what’s under .teamcity is a Maven project. So if we think of it as if it is not a configuration, but an application code, then we can just configure a build configuration that will invoke the Maven goals and run tests and then merge the changes from the branch into master.

      Maybe it is worth using a dedicated repository for the .teamcity configs even.

  3. Avatar

    Jakub Podlesak says:

    November 12, 2019

    Hi Anton,

    Many thanks for the quick response. I think i know how to configure the gate build as code (as i have it running already), the thing is that i do not know how to make sure that build definition updates (that might be part of a pull request) would then be also taken into account by TeamCity when the build is run for an actual pull request. I.e. currently the build definition (.teamcity content) is always taken from the master branch, the build then runs against the pull request branch (all content outside of .teamcity space), and the branch is than merged to the master. This way people could break the build definition if they incorporate new incompatible changes to the .teamcity space (which would pass the gate build for the actual pull request, but could fail the next one). My question is whether the above mentioned scenario is possible due to some misconfiguration on my side, or if it is something that can not be avoided in principle (i know the situation described here resembles this chicken and egg problem). That is the reason why i asked for a blueprint. I can share an example project as requested.



    • Avatar

      Anton Arhipov says:

      November 13, 2019

      Please share the example! Maybe I could come up with a solution that is generic enough and post it as a followup article to this blog post series – everyone would benefit!

  4. Avatar

    Jakub Podlesak says:

    November 12, 2019

    I apologise for posting the same question twice, got a web proxy error initially and thought that my first post was lost, so then submitted the other one only to realise the first post was not actually lost…

Discover more