Kotlin Configuration Scripts: Creating Configuration Scripts Dynamically

This is part three of the five-part series on working with Kotlin to create Configuration Scripts for TeamCity.

  1. An Introduction to Configuration Scripts
  2. Working with Configuration Scripts
  3. Creating Configuration Scripts dynamically
  4. Extending the TeamCity DSL
  5. Testing Configuration Scripts

We saw in the previous post how we could leverage some of Kotlin’s language features to reuse code. In this part, we’re going to take advantage of the fact that we are in fact dealing with a full programming language and not just a limited DSL, to create a dynamic build configuration.

Build Configurations based on parameters

The scenario is the following: we have an HTTP server that we need to test on different platforms with a range of concurrent connections on each one. This generates potentially a lot of different build configurations that we’d need to create and maintain.

Instead of doing this manually, what we can do is write some code to have our Kotlin DSL Configuration Script generate all the different build configurations for us.

Let’s say that we have a list of Operating Systems and a range of concurrent connections for each one. The first thing to do is to create a data class that represents this information

which in essence is like a regular class but provides a series of benefits such as a nicer string representation, equality and copy operations amongst other things.

Now let’s imagine we have the different platforms we want to test on represented as a list of BuildParameters

what we’d like to do, is iterate over this list and create a new build type for each combination. In essence, if our standard Project is

what we want to do is create a new buildType for each entry in the list

Creating a base Build Type

Any parameter passed into buildType needs to be of the type BuildType. This means we’d need to inherit from this class and at the same time provide some parameters to it (BuildParameters)

The code is actually a pretty standard Configuration Script that defines a Gradle build step, has a VCS definition and defines a VCS trigger. What we’ve done is just enhance it somewhat.

To begin with,  we’re using the BuildParameters name property to suffix it to the uuid, extId and name. This guarantees a unique ID per build configuration as well as provides us a name to identify it easily. To make sure the values are properly formatted, we use the TeamCity DSL defined extension function to String, named toExtId() which takes care of removing any forbidden characters.

In the steps definition, we pass in certain parameters to Gradle, which in our case is the number of concurrent connections we want to test with. Obviously, this is just a sample, and the actual data being passed in can be anything and used anywhere in the script.

Finally, we also use the BuildParameters operatingSystem property to define Agent requirements.

Summary

The above is just a sample of what can be done when creating dynamic build scripts. In this case, we created multiple build configurations, but we could just as well have created multiple steps, certain VCS triggers, or whatever else could come in useful. The important thing to understand is that at the end of the day, Kotlin Configuration Script isn’t just merely a DSL but a fully fledged programming language.

In the next part of this series, we’ll see how we can extend the Kotlin Configuration Scripts (hint – it’s going to involve extension functions).

 

 

Posted in Features, How-To's, Tips&Tricks, Uncategorized | 2 Comments

TeamCity 10.0.4 is here

Greetings, everyone!

We have released TeamCity 10.0.4, another update for our latest version. This is mostly a bugfix release addressing over 100 issues, the full list is available in our Release Notes.

As to improvements, starting from version 10.0.4 TeamCity supports Visual Studio 2017.

We recommend that you upgrade, as this version also contains a number of security and performance fixes.

Upgrading is easy, and if you need, you can always roll back to any of the earlier 10.0.x versions.

See the upgrade notes, download TeamCity 10.0.4, and install it on your server to take advantage of the latest improvements.

Happy building!

Posted in Bugfix, Release | Leave a comment

TeamCity Plugin Contest 2016 Winners Announcement

Greetings, everyone!

We’d like to thank all the entrants for making this year’s contest a success!

You have to know that making the final call on the winners was really tough: many of the contestants came up with good quality plugins, some made more than one submission. Several of the plugins were quite simple but well written, some were much more functional. Some covered popular technologies, others implemented integration with less known tools. However, we are sure each plugin will find its audience.

tc_contest_800x320

Every submission was judged against 4 main criteria: usefulness, creativity, completeness, and code quality. The discrepancy between the final candidates was really small, and after a heated discussion, our team of judges came up with the list of winners:

First prize
$3,000 Amazon gift certificate
Winner: reje
Plugin: TeamCity Docker Cloud Plugin

Second prize
$2,000 Amazon gift certificate
Winner: cape
Plugin: Container cloud

Third prize
$1,000 Amazon gift certificate
Winner: grigory.chernyshev
Plugin: Browser notify

Special prize from TeamCity development team
A Trip to JetBrains office in St. Petersburg or Munich
Winner: netwolfuk
Plugin: Debian Package Server

The People’s Choice Award (All Products Pack 1-year subscription) goes to the plugin which got most votes on the contest page:
xUnit test runner 2 by jnyaccuratech.

Congratulations to the winners! We will be contacting you shortly after this announcement. Please ping us if you don’t receive a message from us.

We are grateful to all participants of the competition: all together we received 29 plugins and they are already added to our brand new Plugin Repository. Out of 29, 26 were considered valid and we were happy to grant each of the authors a personal licence to one of the JetBrains IDEs.

Here is the list of these plugins:
Browser notify
Tasks Build Runner
OAuth2 authentication
AlertMePro
Revision properties
xUnit test runner 2
Web parameters
OctoPnP
Agent priority
Telegram Notifier
Agent Auto Authroize
SinCity
Container cloud
Flowdock custom notifier
RoundhousE
Virtual Cloud
Cached Subversion
Config Transformations
Snapshot dependency grouping by project
Gradle Cloud Services
TeamCity Docker Cloud Plugin
Kobalt
rest-runner-teamcity-plugin
GitLab Issues
TeamCity.Swift
Debian Package Server

We are happy with the outcome of the contest, and are definitely going to hold it next year as well. We will add a prize for the best plugin maintenance, and it will go to the person who implements the most considerable improvements to their plugin. It’s just a little more encouragement for you to continue developing your plugins!

Congratulations and happy building!

Posted in Contest, FYI, News & Events | Leave a comment

TeamCity Plugin Contest 2016: Submissions Are Closed, Voting Is Open

The main act of the TeamCity Plugin Contest has come to an end as we closed plugin submissions last night. We would like to thank everyone who spread the word, joined the contest, and spent their valuable time developing new plugins.

Thank you all – we really appreciate your efforts!
linkedin_tccontest_646x220

Of the 112 people registered for the contest, 26 submitted valid plugins, and we were happy to grant them a 1-year subscription for a JetBrains IDE of their choice. All together our participants received 17 IntelliJ IDEA, 5 ReSharper, 2 WebStorm, 1 CLion, and 1 DataGrip subscriptions simply for submitting a valid plugin. Congratulations!

Submissions for the contest are closed, and now the judging and voting for the best plugin begins. You can review all the submitted plugins, and vote for the ones you like best. The author of the most voted plugin will receive the People’s Choice Award!

See the list and vote for plugins

The winners of all the five nominations will be announced on December 15, and nominees will be contacted.

Keep your hopes up, be patient, and remember to pick your favorite plugins. Follow our blog or twitter for the winners announcement.

Thanks again and happy building!

Posted in Contest, News & Events | Leave a comment

Kotlin Configuration Scripts: Working with Configuration Scripts

This is part two of the five-part series on working with Kotlin to create Configuration Scripts for TeamCity.

  1. An Introduction to Configuration Scripts
  2. Working with Configuration Scripts
  3. Creating Configuration Scripts dynamically
  4. Extending the TeamCity DSL
  5. Testing Configuration Scripts

In the first part we saw the basics of Configuration Scripts and how to get started. Now we’ll dive a little deeper into TeamCity’s DSL and see what it provides us in terms of building configuration scripts.

Examining the DSL

The DSL comes in a series of packages with the main one being the actual DSL which is contained in the configs-dsl-kotlin-{version}.jar file. Examining it, we see that it has a series of classes that describe pretty much the entire TeamCity user interface.

main-dls-package

At any time you can navigate to a definition using for instance IntelliJ IDEA, and it will prompt you to download sources or decompile.  The sources are available from the actual TeamCity server your pom.xml file points to, and they’re a great way to see all the potential of the DSL.

The entry point to a project, as we saw in the previous post, is the Project object. From here we can define all the settings such as the basic project properties, the VCS roots, build steps, etc. There are a few basic parameters that should be set:

  • Uuid: it’s the internal ID TeamCity maintains. It is unique across the server. It’s not recommended that this value be changed as TeamCity uses it internally to associate data with the project.  
  • extId: this is the user-friendly ID used in URLs, in the UI, etc. and can be changed if required.
  • parentId: represents the extId of a project where this project belongs, defaulting to the value _Root for top-level projects.
  • name: the name of the project

and optionally

  • description: description for the project

Beyond the above, everything else is pretty much optional. Of course, if that’s the only thing we define, then not much is going to happen during the build process.

Modifying the build process

The code below is an excerpt from the configuration script for the Spek project (parts of the configuration are omitted for brevity). This particular build compiles the code and runs some tests using Gradle.

Let’s extend this build to add a feature to that TeamCity has, Build Files Cleaner also known as Swabra. This build features makes sure that files left by the previous build are removed before running new builds.

We can add it using the features function. As we start to type, we can see that the IDE provides us with completion:

features-completion

The features function takes in turn a series of feature functions, each of which adds a particular feature. In our case, the code we’re looking for is

The resulting code should look like

And this works well. The problem is that, if we want to have this feature for every build configuration, we’re going to end up repeating the code. Let’s refactor it to a better solution.

Refactoring the DSL

What we’d ideally like is to have every build configuration automatically have the Build Files Cleaner feature without having to manually add it. In order to do this, we could introduce a function that wraps every build type with this feature. In essence, instead of having the Project call

we would have it call

For this to work, we’d need to create the following function

Which essentially takes a build type, adds a feature to it, and returns the build type. Given that Kotlin allows top-level functions (i.e. no objects or classes are required to host a function), we can place it anywhere or create a specific file to hold it.

Now let’s extend this function to allow us to pass parameters to our feature, such as the rules to use when cleaning files.

We pass in a list of files which are then passed in as parameter to the feature. The joinToString function allows us to concatenate a list of strings using a specific separator, in our case the carriage return.   

We can improve the code a little so that it only adds the feature if it doesn’t already exist:

Generalizing feature wrappers

The above function is great in that it allows us to add a specific feature to all build configurations. What if we wanted to generalize this so that we could define the feature ourselves? We can do that by passing a block of code to our cleanFiles function, which we’ll also rename to something more generic.

What we’re doing here is creating what’s known as a higher-order function, a function that takes another function as  a function. In fact this is exactly what features, feature, and many of the other TeamCity DSL’s are.

One particular thing about this function, however, is that it’s taking a special function as a parameter, which is an extension function in Kotlin. When passing in this type of parameter, we refer to it as Lambdas with Receivers (i.e. there is a receiver object that the function is applied on).

This then allows us to make the calls to this function in a nice way referencing  feature directly

Summary

In this post we’ve seen how we can modify TeamCity configuration scripts using the extensive  Kotlin-based DSL.  What we really have in our hands is a full programming language along with all the features and power that it provides. We can encapsulate functionality in functions to re-use, we can use higher-order functions  as well as other things that open up many possibilities.

In the next part we’ll see how to use some of this to create scripts dynamically.

 

 

Posted in Design decisions, Features, How-To's, Tips&Tricks | Leave a comment

TeamCity Plugin Contest 2016: One Week and Counting

Greetings everyone!

For one more week TeamCity Contest is accepting new plugins: you have until December 6, 2016 to submit your creations!

fb_tccontest_1200x628

Make sure you use this opportunity to win up to $3,000 in Amazon gift certificates, a trip to a JetBrains office, or the All Products Pack 1-year subscription. Oh, and remember that a 1-year subscription for a JetBrains IDE of your choice is guaranteed for any accepted plugin!

This is the final call for submissions – the results will be announced on December 15, 2016.

Hats off to our creative users who have already sent us their plugins!

All accepted submissions are published to our new TeamCity plugin repository. Follow our blog for the news and check the #TeamCityContest hashtag on Twitter!

pluginandwin

Our forum and tracker are awaiting your feedback!

Happy developing and building!

Posted in Contest, FYI, News & Events | Leave a comment

Webinar Recording: Using TeamCity 10 to Version Control Your Continuous Delivery Pipeline

A couple of weeks ago, ClearMeasure, our partner, held a webinar titled “Using TeamCity 10 to Version Control Your Continuous Delivery Pipeline”. In this webinar, Jeffrey Palermo (Managing Partner and CEO of Clear Measure) and Yujie Wu (software engineer at Clear Measure) demonstrated the high points of TeamCity 10 and dove into a fully-configured continuous delivery implementation using code-based build configurations.

The webinar focuses on the following points:
– How to convert your existing TeamCity builds into versioned code-based builds
– Demonstration of a fully operable CD pipeline using TeamCity 10 with downloadable code, build config, and everything needed to replicate it on your own

Here is the recording of that webinar, along with the select Q&A from the session:

Q:  Why use Build Chains with one step build configurations vs multiple steps in a single build configuration?
A:  There can be a variety of reasons. First, build chain is not necessarily a pipeline, it’s a graph. Build chain allows to run tasks in parallel, if possible. Second, build chain allows to not rerun the build if there were no changes. And actually there are many other reasons as well, such as better visibility into steps and so on. Sometimes multiples steps are better, sometimes build chains are more convenient.

Q:  Would Kotlin-based build configurations replace the concept of TeamCity build configuration templates?
A:  No, templates will remain. DSL is much more flexible, but templates are still handy in many cases.

Q:  So, you would choose one or the other depending on your use case?
A:  Exactly. In case when templates are not enough, you can go with DSL.

Q:  Are the artifact dependencies strictly necessary for the “promote” build configs? I have the same type of build chain setup, but my “promote” build configs only have the snapshot dependency set. Wouldn’t Octopus Deploy restrict what release gets promoted?
A:  Artifact dependency is the way to share build results with other builds. For example, you have built and tested parts separately and on the last step you combine them into a single package.

Posted in Features, Partner, Webinar | Leave a comment

Kotlin Configuration Scripts: An Introduction

This is a five part series on working with Kotlin to create Configuration Scripts for TeamCity.

  1. An Introduction to Configuration Scripts
  2. Working with Configuration Scripts
  3. Creating Configuration Scripts dynamically
  4. Extending the TeamCity DSL
  5. Testing Configuration Scripts

With TeamCity 10, we brought the ability to define configurations using Kotlin as opposed to using XML or the User Interface.

This provides several benefits amongst which are

  • A human-readable and editable configuration script, not relying on the user interface
  • Ability to do diffs on changes to a configuration script
  • Ability to check in your configuration scripts alongside your source code1
  • Ability to dynamically create configuration scripts, as well as test them

Why Kotlin?

You might ask why Kotlin and not some other language? For those not familiar with Kotlin, it is a statically-typed language developed by us at JetBrains and open sourced under the Apache 2 license. It targets the JVM, JavaScript (and we also have native in the works). It’s designed to be a pragmatic language that cuts down boilerplate code, remains expressive, and facilitates tooling. Kotlin provides a series of features that allow the creation of DSL’s (Domain Specific Languages) and TeamCity is a perfect fit for this purpose. Kotlin enables the benefits we’ve outlined and we’ll see.

Why not something more common such as YAML? Based on our experience, we believe that while YAML is great for the simpler setups, at some point it does fall short, and can lose clarity when it comes to defining more complex configuration scripts. We wanted to provide a solution that works for the simplest to the most demanding of scenarios, and that’s why we’ve gone with Kotlin. It’s important to understand though that we’re not providing just a DSL for creating configuration scripts. Kotlin is a programming language and as such we have the ability to write any kind of code in our configuration scripts (which, of course, like anything can be also abused). This enables many scenarios such as those we’ll see when creating dynamic configurations.

What is needed

Given Kotlin is a proper language, you might be wondering what tooling is required to edit configuration scripts. In principle any editor will do. Kotlin is built so that it can be used with any editor or IDE. In fact we ship support for not only IntelliJ IDEA (both Ultimate and the free OSS community editions), but also Eclipse and NetBeans. If Vim or another editor is your thing, you can also use that along with a command line compiler.

For the purpose of this blog post series, we’ll be using IntelliJ IDEA.

Creating a first script

While we can start with a completely blank script, the easiest way to create a Kotlin build script is to take an existing TeamCity project and either

  • Download it as a Kotlin Script
  • Enable Versioned Settings

The first is a great way to start playing with Kotlin configuration scripts and get a feel for them. The second option not only provides us with the configuration scripts checked in to source control, but it’s actually a required step for us to use Kotlin build scripts in production.

The recommended approach is to use the second option once we’re ready to enable Kotlin scripting in our projects. The first option, as mentioned, is great for discovering how things work and getting familiar with Kotlin configuration scripts.

Download Settings as a Kotlin Script

In order to see what a build configuration looks like in Kotlin, simply Edit Project Settings on the selected project and click on the Actions menu, picking the entry Download Configuration Script in Kotlin format

download-kotlin-settings

This downloads a zip file that contains a Maven project which can be obtained in IntelliJ IDEA. Note that the folder it contains is prefixed with a dot, which indicates it’s hidden under MacOS. The idea is that this folder can essentially be placed in version control at some point (TeamCity marks its configuration files as hidden).

Enable Versioned Settings

The second option, which is required for us to actually use Kotlin configuration scripts, is to enable Versioned Settings. This is done under Versioned Settings and selecting Enable, and Kotlin as the file format.

versioned-settings

 

As soon as we activate this, the TeamCity UI will no longer allow any changes and all the configuration will be stored in version control and modifications have to take place via the build script.

These script files will have the same folder layout as that of the downloaded zip file (first option). For instance, the following are the files checked in to version control for the Joda-Time project. We can see that once again it’s a Maven project containing a series of Kotlin files (.kt). 

Opening the script in IntelliJ IDEA

Once we have the configuration script (be it as a zip file or checked in to version control on enabled versioned settings), we can open it in IntelliJ IDEA, and Maven will pull down the correct dependencies. Our project layout should look like the one below

project-structure

Don’t feel overwhelmed by the number of files in there. In fact, TeamCity really only needs one file, which is settings.kts. Let’s examine each file in more detail.

settings.kts

kts is a Kotlin Script file, different from a Kotlin file (.kt) in that it can be run as a script. As mentioned above, all the code relevant to the TeamCity configuration could be stored in this single file, but it is divided into several files to provide a better separation of concerns.

This file contains usually two lines

settings

version indicates the TeamCity version, and project() is the main entry point to the configuration script. It represents a function call, which takes as parameter a Project, representing the entire TeamCity project.

The parameter GitExtensions.Project is an object in Kotlin. Think of objects in Kotlin as you would in JavaScript. Or to compare it to Java, they’d be a Singleton, a single instance of a Kotlin class. While Kotlin script could work with singletons, having a first-class support for objects, makes things much easier.

In this case, Project is the name of the actual object, and GitExtensions is the package name.

Project.kt

This file contains the GitExtensions.Project object, and is where all the magic happens. If you look at the script layout, it’s basically a code representation of the build steps we’re accustomed to seeing the TeamCity user interface

project

which would correspond to the following entries in the UI, in addition to VCS roots and Build Types.

settings-ui

GitExtensions_HttpsGithubComJetbrainsGitextensions.kt

This object defines the VCS root configuration. It’s important to note that we could have just placed all this information directly in the code above, as opposed to making the call vcsRoot(GitExtensions_…).  However, the advantage to doing this, as we’ll see, is not only to make the code cleaner and separate concerns, but also to provide  reusability.

vcs-settings-root

GitExtensions_Main.kt

Finally, we have the actual meat of where the build happens. This object defines the build steps, failure conditions, and everything else you’d come to expect for a build configuration in TeamCity

build-type

Summary

In this first part we’ve seen the components of a TeamCity configuration script. In the next part we’ll dive a little deeper into the DSL, modify the script, and see some of the benefits that Kotlin and IntelliJ IDEA already start providing us in terms of guidance via code assistants.

[1] While enabling version control storage for settings has been available since version 9, it was only available in XML format. TeamCity 10 brings Kotlin to this.

Posted in Features, FYI, How-To's, Tips&Tricks | 1 Comment

TeamCity 10.0.3 has arrived

Greetings, everyone!

We are happy to announce the arrival of TeamCity 10.0.3 (build #42434), a bugfix release containing over 200 improvements.

We’ll mention a few of them:

– We added a filter by agent pool to the build time report (this can help you analyze which project or build configuration consumes most of the pool resources)

– If you want to experiment with project settings in Kotlin-based DSL, you can now do it without disabling the UI: use the Actions menu in the project administration area to download the settings in this format as a .zip archive and use them in your sandbox

– TeamCity now properly detects Visual Studio 15 and Java 9 on agents

– We finally open-sourced the Dockerfiles used to prepare TeamCity server and agent images (see TW-46505 for details)

This version introduces a number of performance and security fixes, so it is recommended you upgrade your TeamCity to the latest version.

The full list of fixes is available in our Release notes.

Check the upgrade notes and download the latest version: as usual, all the 10.0.x builds use the same data format, so you can downgrade any time you like.

Happy building!

Posted in Bugfix, FYI, Release | 1 Comment

Build, test and deploy .NET Core projects with TeamCity

teamcity-net-coreThe .NET Core development platform is the next wave of .NET ecosystem evolution which brings cross-platform targets for your existing or new products. Microsoft ships .NET Core SDK packages with a runtime for supported platforms and command line tools (.NET CLI) to build, test, and pack such projects.

The .NET Core support plugin for TeamCity aims to bring streamline support for .NET Core CLI by providing .NET Core build steps, CLI detection on the build agents, and auto-discovery of build steps in your repository.

Installation

The plugin is available for download from the TeamCity plugin gallery and can be installed as an additional TeamCity plugin. Next the .NET Core SDK has to be installed on your build agent machines. The plugin uses the PATH environment variable to determine the .NET CLI tools location, but you can override it by specifying the DOTNET_HOME environment variable as follows: DOTNET_HOME=C:\Program Files\dotnet\

You can check that plugin has been properly initialized by navigating to the build agent parameters:

net-core-parameters

Building and testing .NET Core projects

The .NET Core plugin supports auto-discovery of build steps, and usually all you need is to use the Create project from URL wizard:

net-core-build-steps

If you want to tune some parameters in your build steps, the .NET Core plugin provides details about configuration options and hints at their possible values:

net-core-step

Specify project build version

While building and creating project packages, the .NET CLI tools use the version number defined in the project.json file. If you want to provide a custom build number, you can update your project.json by leveraging the File Content Replacer build feature as follows:

net-core-version

Here you need to specify the target project.json files to process, fill in (\{[^{]*\"version\"\s*:\s*\")([^\"]+)(\") in the Find what and $1%build.number%$3 in the Replace with fields. Then, after executing the vcs checkout, the File Content Replacer will set the version, for instance %build.number% in the previous example.

Note: Currently .NET CLI tools support projects with project.json-based structure, but the upcoming versions will rely on the new csproj structure.

Testing .NET Core projects

The xUnit framework provide dotnet test runner which report the information about test results on the fly:

net-core-tests

The NUnit test runner does not support it out of the box, but you can vote for the following issue. If you need any other test framework support, please leave a comment in the related issue.

Deploying .NET Core projects

.NET Core libraries

Your projects can contain a number of shared libraries which can be reused in dependent project builds. Instead of artifact dependencies, it is recommended that you use the build-in TeamCity NuGet feed. In this the specified build artifacts paths need to include the produced NuGet packages.

Tip: You can use the NuGet Dependency Trigger to start a build when a package has been updated in some nuget feed.

Publishing ASP.NET Core projects

Such projects can be deployed to a wide range of platforms. As a result of executing the dotnet publish build step, you will receive the published package data in your file system. Then you need to decide how to deploy it to the target system.

For Windows servers, the Web Deploy utility can be used. Alternatively, you can deploy the package with the auto-generated %profile%-publish.ps1 or default-publish.ps1 PowerShell scripts. More details on how to provide proper arguments to these scripts can be found in the Web Publishing documentation.

For other server platforms you can publish package by leveraging TeamCity Deployer build steps like FTP/SSH/SMB.

Feedback

You are welcome to try the plugin for TeamCity, and your feedback will be really appreciated, so please share your thoughts in the comments to this blog post, in the forum or issue tracker.

Happy building modern .NET Core projects!

Posted in Features, Tips&Tricks | Tagged , , | 27 Comments