Kotlin Configuration Scripts: An Introduction

This is a five part series on working with Kotlin to create Configuration Scripts for TeamCity.

  1. An Introduction to Configuration Scripts
  2. Working with Configuration Scripts
  3. Creating Configuration Scripts dynamically
  4. Extending the TeamCity DSL
  5. Testing Configuration Scripts

With TeamCity 10, we brought the ability to define configurations using Kotlin as opposed to using XML or the User Interface.

This provides several benefits amongst which are

  • A human-readable and editable configuration script, not relying on the user interface
  • Ability to do diffs on changes to a configuration script
  • Ability to check in your configuration scripts alongside your source code1
  • Ability to dynamically create configuration scripts, as well as test them

Why Kotlin?

You might ask why Kotlin and not some other language? For those not familiar with Kotlin, it is a statically-typed language developed by us at JetBrains and open sourced under the Apache 2 license. It targets the JVM, JavaScript (and we also have native in the works). It’s designed to be a pragmatic language that cuts down boilerplate code, remains expressive, and facilitates tooling. Kotlin provides a series of features that allow the creation of DSL’s (Domain Specific Languages) and TeamCity is a perfect fit for this purpose. Kotlin enables the benefits we’ve outlined and we’ll see.

Why not something more common such as YAML? Based on our experience, we believe that while YAML is great for the simpler setups, at some point it does fall short, and can lose clarity when it comes to defining more complex configuration scripts. We wanted to provide a solution that works for the simplest to the most demanding of scenarios, and that’s why we’ve gone with Kotlin. It’s important to understand though that we’re not providing just a DSL for creating configuration scripts. Kotlin is a programming language and as such we have the ability to write any kind of code in our configuration scripts (which, of course, like anything can be also abused). This enables many scenarios such as those we’ll see when creating dynamic configurations.

What is needed

Given Kotlin is a proper language, you might be wondering what tooling is required to edit configuration scripts. In principle any editor will do. Kotlin is built so that it can be used with any editor or IDE. In fact we ship support for not only IntelliJ IDEA (both Ultimate and the free OSS community editions), but also Eclipse and NetBeans. If Vim or another editor is your thing, you can also use that along with a command line compiler.

For the purpose of this blog post series, we’ll be using IntelliJ IDEA.

Creating a first script

While we can start with a completely blank script, the easiest way to create a Kotlin build script is to take an existing TeamCity project and either

  • Download it as a Kotlin Script
  • Enable Versioned Settings

The first is a great way to start playing with Kotlin configuration scripts and get a feel for them. The second option not only provides us with the configuration scripts checked in to source control, but it’s actually a required step for us to use Kotlin build scripts in production.

The recommended approach is to use the second option once we’re ready to enable Kotlin scripting in our projects. The first option, as mentioned, is great for discovering how things work and getting familiar with Kotlin configuration scripts.

Download Settings as a Kotlin Script

In order to see what a build configuration looks like in Kotlin, simply Edit Project Settings on the selected project and click on the Actions menu, picking the entry Download Configuration Script in Kotlin format

download-kotlin-settings

This downloads a zip file that contains a Maven project which can be obtained in IntelliJ IDEA. Note that the folder it contains is prefixed with a dot, which indicates it’s hidden under MacOS. The idea is that this folder can essentially be placed in version control at some point (TeamCity marks its configuration files as hidden).

Enable Versioned Settings

The second option, which is required for us to actually use Kotlin configuration scripts, is to enable Versioned Settings. This is done under Versioned Settings and selecting Enable, and Kotlin as the file format.

versioned-settings

 

As soon as we activate this, the TeamCity UI will no longer allow any changes and all the configuration will be stored in version control and modifications have to take place via the build script.

These script files will have the same folder layout as that of the downloaded zip file (first option). For instance, the following are the files checked in to version control for the Joda-Time project. We can see that once again it’s a Maven project containing a series of Kotlin files (.kt). 

Opening the script in IntelliJ IDEA

Once we have the configuration script (be it as a zip file or checked in to version control on enabled versioned settings), we can open it in IntelliJ IDEA, and Maven will pull down the correct dependencies. Our project layout should look like the one below

project-structure

Don’t feel overwhelmed by the number of files in there. In fact, TeamCity really only needs one file, which is settings.kts. Let’s examine each file in more detail.

settings.kts

kts is a Kotlin Script file, different from a Kotlin file (.kt) in that it can be run as a script. As mentioned above, all the code relevant to the TeamCity configuration could be stored in this single file, but it is divided into several files to provide a better separation of concerns.

This file contains usually two lines

settings

version indicates the TeamCity version, and project() is the main entry point to the configuration script. It represents a function call, which takes as parameter a Project, representing the entire TeamCity project.

The parameter GitExtensions.Project is an object in Kotlin. Think of objects in Kotlin as you would in JavaScript. Or to compare it to Java, they’d be a Singleton, a single instance of a Kotlin class. While Kotlin script could work with singletons, having a first-class support for objects, makes things much easier.

In this case, Project is the name of the actual object, and GitExtensions is the package name.

Project.kt

This file contains the GitExtensions.Project object, and is where all the magic happens. If you look at the script layout, it’s basically a code representation of the build steps we’re accustomed to seeing the TeamCity user interface

project

which would correspond to the following entries in the UI, in addition to VCS roots and Build Types.

settings-ui

GitExtensions_HttpsGithubComJetbrainsGitextensions.kt

This object defines the VCS root configuration. It’s important to note that we could have just placed all this information directly in the code above, as opposed to making the call vcsRoot(GitExtensions_…).  However, the advantage to doing this, as we’ll see, is not only to make the code cleaner and separate concerns, but also to provide  reusability.

vcs-settings-root

GitExtensions_Main.kt

Finally, we have the actual meat of where the build happens. This object defines the build steps, failure conditions, and everything else you’d come to expect for a build configuration in TeamCity

build-type

Summary

In this first part we’ve seen the components of a TeamCity configuration script. In the next part we’ll dive a little deeper into the DSL, modify the script, and see some of the benefits that Kotlin and IntelliJ IDEA already start providing us in terms of guidance via code assistants.

[1] While enabling version control storage for settings has been available since version 9, it was only available in XML format. TeamCity 10 brings Kotlin to this.

Posted in Features, FYI, How-To's, Tips&Tricks | Leave a comment

TeamCity 10.0.3 has arrived

Greetings, everyone!

We are happy to announce the arrival of TeamCity 10.0.3 (build #42434), a bugfix release containing over 200 improvements.

We’ll mention a few of them:

– We added a filter by agent pool to the build time report (this can help you analyze which project or build configuration consumes most of the pool resources)

– If you want to experiment with project settings in Kotlin-based DSL, you can now do it without disabling the UI: use the Actions menu in the project administration area to download the settings in this format as a .zip archive and use them in your sandbox

– TeamCity now properly detects Visual Studio 15 and Java 9 on agents

– We finally open-sourced the Dockerfiles used to prepare TeamCity server and agent images (see TW-46505 for details)

This version introduces a number of performance and security fixes, so it is recommended you upgrade your TeamCity to the latest version.

The full list of fixes is available in our Release notes.

Check the upgrade notes and download the latest version: as usual, all the 10.0.x builds use the same data format, so you can downgrade any time you like.

Happy building!

Posted in Bugfix, FYI, Release | 1 Comment

Build, test and deploy .NET Core projects with TeamCity

teamcity-net-coreThe .NET Core development platform is the next wave of .NET ecosystem evolution which brings cross-platform targets for your existing or new products. Microsoft ships .NET Core SDK packages with a runtime for supported platforms and command line tools (.NET CLI) to build, test, and pack such projects.

The .NET Core support plugin for TeamCity aims to bring streamline support for .NET Core CLI by providing .NET Core build steps, CLI detection on the build agents, and auto-discovery of build steps in your repository.

Installation

The plugin is available for download from the TeamCity plugin gallery and can be installed as an additional TeamCity plugin. Next the .NET Core SDK has to be installed on your build agent machines. The plugin uses the PATH environment variable to determine the .NET CLI tools location, but you can override it by specifying the DOTNET_HOME environment variable as follows: DOTNET_HOME=C:\Program Files\dotnet\

You can check that plugin has been properly initialized by navigating to the build agent parameters:

net-core-parameters

Building and testing .NET Core projects

The .NET Core plugin supports auto-discovery of build steps, and usually all you need is to use the Create project from URL wizard:

net-core-build-steps

If you want to tune some parameters in your build steps, the .NET Core plugin provides details about configuration options and hints at their possible values:

net-core-step

Specify project build version

While building and creating project packages, the .NET CLI tools use the version number defined in the project.json file. If you want to provide a custom build number, you can update your project.json by leveraging the File Content Replacer build feature as follows:

net-core-version

Here you need to specify the target project.json files to process, fill in (\{[^{]*\"version\"\s*:\s*\")([^\"]+)(\") in the Find what and $1%build.number%$3 in the Replace with fields. Then, after executing the vcs checkout, the File Content Replacer will set the version, for instance %build.number% in the previous example.

Note: Currently .NET CLI tools support projects with project.json-based structure, but the upcoming versions will rely on the new csproj structure.

Testing .NET Core projects

The xUnit framework provide dotnet test runner which report the information about test results on the fly:

net-core-tests

The NUnit test runner does not support it out of the box, but you can vote for the following issue. If you need any other test framework support, please leave a comment in the related issue.

Deploying .NET Core projects

.NET Core libraries

Your projects can contain a number of shared libraries which can be reused in dependent project builds. Instead of artifact dependencies, it is recommended that you use the build-in TeamCity NuGet feed. In this the specified build artifacts paths need to include the produced NuGet packages.

Tip: You can use the NuGet Dependency Trigger to start a build when a package has been updated in some nuget feed.

Publishing ASP.NET Core projects

Such projects can be deployed to a wide range of platforms. As a result of executing the dotnet publish build step, you will receive the published package data in your file system. Then you need to decide how to deploy it to the target system.

For Windows servers, the Web Deploy utility can be used. Alternatively, you can deploy the package with the auto-generated %profile%-publish.ps1 or default-publish.ps1 PowerShell scripts. More details on how to provide proper arguments to these scripts can be found in the Web Publishing documentation.

For other server platforms you can publish package by leveraging TeamCity Deployer build steps like FTP/SSH/SMB.

Feedback

You are welcome to try the plugin for TeamCity, and your feedback will be really appreciated, so please share your thoughts in the comments to this blog post, in the forum or issue tracker.

Happy building modern .NET Core projects!

Posted in Features, Tips&Tricks | Tagged , , | 20 Comments

Hosted TeamCity for Open Source – A New Home

Back in 2008, with the release of TeamCity 3.1, James Kovacs, Brendan Tompkins and a few others contacted us in order to provide a hosted TeamCity instance for OSS projects in .NET. That ultimately became the TeamCity at CodeBetter site.

For quite a few number of years, the service was maintained and hosted by volunteers from CodeBetter and the last years by Anne Epstein, who has been doing a wonderful job of single-handedly managing new projects, etc.

JetBrains’ involvement in the project was merely limited to providing TeamCity licenses and monetary support for the server, until around 2014. Since then we’ve been slightly more involved in terms of TeamCity administration. However server maintenance and other aspects were out of our area of responsibility.

While doing this, we were also in parallel conversations of providing a more reliable and faster service (and servers) to OSS projects and somehow consolidating .NET projects with what we’ve been offering for projects on TeamCity at JetBrains

Unfortunately this past September, the hardware died  and did so badly. Luckily we managed to rescue most of the projects and data and continue with the migration process to our own servers. For a complete list of what was ported and what was not please see What was Saved below.

Hosting the OSS projects

If your project was one of those migrated from CodeBetter, then you should already have it available on the new server. If any issues arise, please ping us at teamcity-support@jetbrains.com.

We are conservatively adding new projects, but at a slower rate than with CodeBetter in that we want to make sure we can cater to a correct level of service that is required. If you’re looking at hosting a new project, please reach out to us at teamcity-support@jetbrains.com.

What was saved

  • Projects and their settings. Project and build configuration IDs used in URLs are also left as they were except for conflicting cases where the “cb_” prefix was added to the IDs.
  • Users with non-view roles. The majority of user accounts with special permissions from teamcity.codebetter.com continue to work with the same credentials. Some of the users who had the same usernames on teamcity.jetbrains.com server, need to use their teamcity.jetbrains.com credentials to login. Users with no special permissions: please create new account if you do not already have one.
  • History of builds and changes
  • 15% of the build logs and artifacts
  • The same “JetBrains” agents are connected to the server

Some changes

  • The server address. http://teamcity.codebetter.com/ shows a move notice page, sub-URLs are redirected to teamcity.jetbrains.com server
  • Internal build- and change-level URLs (numeric IDs are remapped to new values)
  • Majority of the build logs and build artifacts are not present. It should be possible to rerun an important history build to get the same revision artifacts.
  • Old “CodeBetter” agents are no longer connected to the server
  • Several plugins are no longer available: GitHub issues, TeamCity.Node, Bazaar, CMake, GitHub status (use “Commit status publisher” instead)
  • New “from” in Email and Jabber notifications sent out by the server

Acknowledgements

We really appreciate all the effort that Anne Epstein put into maintaining the CodeBetter. We also want to thank James, Brendan, and all those involved throughout these years in bringing TeamCity to the .NET Open Source world.

Posted in Blogroll, FYI, News & Events | Leave a comment

TeamCity 10 GitHub-Related Improvements

Configuring TeamCity to use GitHub as the source code repository has always been easy, especially since the feature ‘create from URL‘ was first introduced. TeamCity 10 has brought a number of improvements related to integration with GitHub, which are worth a special mention.

Project-level GitHub Connection in TeamCity

We’ve been gradually improving the integration, and in TeamCity 10 you can now set up a connection to GitHub on the project level:

connectionproject

Once the connection is set up, you can create a project or a build configuration pointing to a GitHub repository:connectionprojectlist

Besides, with the connection configured, a small GitHub icon becomes active in several places where a repository URL can be specified: create project from URL, create Git VCS root, GitHub issue tracker, create create VCS root from URL:

createvcsroot

On clicking the icon, TeamCity will list GitHub repositories available to the user.

Configuring a GitHub connection is useful for the organization administrator who can create a parent project and configure a connection to GitHub there once; thus, all the TeamCity users of the organization will see a list of GitHub repositories URLs in the TeamCity Web UI. It makes setting up a subproject, a Git VCS root, or a GitHub issue tracker extremely easy.

GitHub Issue Tracker Integration in TeamCity

Built-in integration with GitHub issue tracker was also introduced in TeamCity 10. It is configured on the dedicated page of the project settings. If a project level connection to GitHub is configured, you can simply select the repository URL from the list available on clicking the GitHub icon:

tracker

If no connection is configured, the URL can be specified manually.

The rest is easy – select the type of authentication and provide the required information, tell TeamCity which strings should be recognized as references to issues: for GitHub, the regex syntax is used, e.g. #(\d+). TeamCity will resolve the issue number mentioned in a VCS comment and will display a link to this issue in the Web UI (e.g. on the Changes page, Issues tab of the build results page).

TeamCity Build Status in GitHub

Developers find it handy to view the status of their commits right in the repository. Earlier there were several external plugins allowing you to publish the TeamCity build status on GitHub, and in TeamCity 10 we delivered a bundled build feature, Commit Status Publisher, which automatically attaches the build status to your GitHub pull requests:

commitstatuspub

Besides, using the TeamCity REST API, you can publish the status icon of your TeamCity build in the README for your repository:

statuspic

TeamCity Commit Hooks

All the above-mentioned improvements are bundled with TeamCity 10, but we should also mention the TeamCity Commit Hooks Plugin, not bundled with TeamCity yet. This plugin is compatible with TeamCity 10 or later. Its task is to install webhooks for GitHub repositories specified in TeamCity VCS roots.
Installed GitHub webhooks greatly decrease the time required for TeamCity to detect a change. As an additional benefit, webhooks reduce the load on both the TeamCity server and the GitHub server.

 

Take the advantage of the TeamCity-GitHub improved integration and let these features make your experience of Continuous Integration & Delivery with GitHub and TeamCity nice and smooth.

Posted in Features, FYI, How-To's, Tips&Tricks | Leave a comment

TeamCity 2016 Plugin Contest Announcement

Greetings everyone!

We are pleased to announce TeamCity Plugin Contest 2016 lasting from October 6 to December 6! This is your chance not only to provide a missing integration or some cool feature for TeamCity, but also to get a guaranteed reward for participation and win much bigger prizes!

Create your own plugin and submit it to us by December 6, 2016! Our team of judges will evaluate all the submissions and grant the prizes. The results will be announced on December 15, 2016.

New to TeamCity plugin development and do not know where to start? Check out our Getting Started guide.

The countdown has started! We’ll keep you posted:  follow our blog and  check our Twitter account, the #TeamCityContest hashtag.

And one more thing: your plugin submissions will appear in our new experimental TeamCity plugin repository which has just been opened to the public.

We love getting your feedback – our forum and tracker are at your disposal!

Happy developing and building!

Posted in Contest, FYI, News & Events | Comments Off on TeamCity 2016 Plugin Contest Announcement

Installing GitHub Webhooks from TeamCity

Greetings, everyone!

Since TeamCity 10.0 it has become possible to use commit hooks with a TeamCity server. Now, when a VCS change is detected via a commit hook, TeamCity automatically increases the VCS repository polling interval, reducing the load on both the TeamCity server and VCS repository. And obviously the presence of the hook greatly decreases the time needed to detect a change.

Unfortunately, installation and configuration of commit hooks is not an easy task and for on-premises VCS repositories it requires administration skills. At the same time, popular VCS hostings such as GitHub, support installation of commit hooks via their REST API, and installation of commit hooks for them can be a lot simpler, provided that we use such an API from TeamCity…

So today, we’d like to announce one more TeamCity plugin whose task is to install and maintain GitHub commit hooks (do not worry, we plan to add support for other VCS hosting services too).

The plugin supports both GitHub.com and GitHub Enterprise. The plugin does not install a webhook automatically for GitHub.com because a webhook requires a connection from GitHub.com to the TeamCity server, and in the majority of cases, when TeamCity is installed in the intranet, such a connection is blocked by a firewall. In case of GitHub Enterprise, the plugin will install a webhook automatically for any TeamCity project created from a URL or via GitHub integration.

The plugin works with the GitHub REST API and has to make API calls to GitHub on behalf of the current user, so it requires a GitHub connection configured in the project or its parent.

The plugin is quite simple, basically it does three things:

  • It shows a suggestion to install a GitHub webhook if it finds a GitHub repository in a project without such a webhook:webhook_suggestion
  • It provides a new action in the project actions menu for webhook installation enabling you to install or reinstall a webhook at any time: project_actions
  • It checks the status of all of the installed webhooks and raises a warning via the health report if some problem is detected:webhook_problem

The plugin is open source, distributed under the Apache 2.0 license. Most of the code is written in Kotlin. The source code is published on GitHub.

The plugin relies on the new TeamCity API and will only work with TeamCity 10.0 and later. Download the plugin, install it on your TeamCity server and try the plugin. We’ll appreciate your feedback!

Happy building!

Posted in Features, How-To's, News & Events, Tips&Tricks | Leave a comment

TeamCity 10.0.2 is out today

Greeting, everyone!

TeamCity 10.0.2, an update fixing over 200 issues, is available now.

Among the other improvements, build 42234 includes quite a few NuGet-related fixes and several enhancements in the Commit Status Publisher plugin; for the full list, see our release notes.

This bugfix release uses the same data format as the other 10.0.x builds, which means you are free to upgrade/downgrade within the latest major version, so just download TeamCity 10.0.2 and upgrade!

Happy upgrading and building!

Posted in Bugfix, Release | Leave a comment

Webinar recording: Building Scalable, Repeatable Infrastructure in the Cloud

The recording of our August 30 webinar, Building Scalable, Repeatable Infrastructure in the Cloud, is now available on the JetBrains YouTube channel.

In this webinar, Paul Stack helps you design the deployment of your applications into AWS. He demonstrates, using Terraform, that the practices of Infrastructure as Code allow the most scalable and repeatable infrastructure. His demo is be based on deploying TeamCity in AWS using a hosted PostgreSQL database.

 

Posted in Partner, Webinar | Leave a comment

August 30 live webinar: Building Scalable, Repeatable Infrastructure in the Cloud

Join us Tuesday, August 30th, 16:00 – 17:00 CEST (10:00 – 11:00 EDT) for our free live webinar, Building Scalable, Repeatable Infrastructure in the Cloud with Paul Stack.

In this webinar, Paul Stack will help you design the deployment of your applications into AWS. He will demonstrate, using Terraform, that the practices of Infrastructure as Code will allow the most scalable and repeatable infrastructure. His demo will be based on deploying TeamCity in AWS using a hosted PostgreSQL database.

Space is limited, please register now. There will be an opportunity to ask questions during the webinar.

About the presenter:

paulSRightPaul Stack is a London based infrastructure coder and has spoken at various events throughout the world about his passion for continuous integration, continuous delivery and good operational procedures and why they should be part of what developers and system administrators do on a day to day basis. He believes that reliably delivering software is as important as its development.

Paul’s passions are the DevOps and Continuous Delivery movements and how they help the entire business and its customers.

Posted in Partner, Webinar | Leave a comment