TeamCity brings Azure Resource Manager support

arm-logo

Microsoft has recently introduced a new way to deploy cloud resources, so currently there exist two Azure deployment models supported by TeamCity as two separate plugins.

The earlier Azure deployment model is in the feature freeze state and is now called Azure Classic. The same goes for the existing TeamCity Azure support, now called TeamCity Azure Classic plugin.

Azure Resource Manager is the new Azure deployment model, which has some advanced features and is now becoming popular. More and more users want to deploy cloud agents with it, so today we are glad to announce a new plugin, the Azure Resource Manager plugin for TeamCity.

Plugin Description

The plugin for TeamCity starts new cloud build agents from a generalized VHD image when all of the regular build agents are busy. It uses resource groups to deploy build agent resources and supports authentication with an Azure Active Directory (AD) application.

While starting a new virtual machine, the plugin creates a new resource group and deploys required resources into it. When a cloud agent is terminating, the resource group with the allocated resources is removed.

The latest plugin version is available for download to the TeamCity 9.1.x users as a separate plugin which can be installed manually.

Azure Authentication Settings

The plugin accesses the Azure Resource Manager by leveraging an Azure AD application. To create a new application, follow the steps described in the Azure documentation. Then open new portal, navigate to Subscriptions, select the target subscription and assign a Contributor role to your Azure AD application.

Note: To find your application, type the application name in the users search box, because originally that list contains only usernames, not applications.

Virtual Machines Preparation

The plugin supports generalized VHD images to start new TeamCity cloud build agents. Perform the steps below to prepare VHD images.

New Image Creation

Create a new virtual machine with the Resource Manager via the new portal or cli tools, and configure it properly. Then fulfill generalization according to the Azure documentation for Linux and Windows virtual machines. As a result, you will get a VHD image in your blob storage.

Migrating Specialized Classic VM Image

If you previously used a specialized image or Start/Stop behaviour for a cloud build agent, you must capture it now. To do that, prepare a virtual machine by removing temporary files and then make a new image capture. Please refer to the Azure documentation for Linux and Windows machines. Then follow the steps below to migrate a generalized classic VM image.

Migrating Generalized Classic VM Image

If you already have a generalized VHD image located in the classic storage account, just copy it to a storage account created with the Resource Manager. To do that, use Microsoft Azure Storage Explorer.

Network Settings

To start new virtual machines, ensure that you have a Virtual Network and Subnet created in the target Azure region. If you already have virtual machines deployed with the Resource Manager, you can use them; otherwise you must create a new one and configure it as follows:

  1. Open the Virtual Network creation widget on the new portal and press Create.
  2. Fill in the settings and select the Resource Group located in your target Azure region.
  3. Press Create and wait for the deployment completion.

Note: To access your virtual machines by a public IP address, please ensure that you have properly configured security group rules for your Subnet.

Configuring Agent Cloud Profile in TeamCity

Open TeamCity Agent Cloud configuration and create a new Azure Resource Manager profile. Fill in the AD application credentials from the classic portal: copy the tenant id, client id values, and create a new client secret. Then select your subscription and target region to deploy new TeamCity build agent resources:

arm-credentials

Configuring Build Agent Images

Then add a new agent image in the cloud profile by pressing the Add image button and set configuration parameters:

arm-image

Source image is a VHD blob URL of generalized Virtual Machine. Please note that currently the Azure Resource Manager allows creating OS disk images only in the original storage account where the source VHD image is stored. More details are available in the Azure forum request.

Name prefix will be used as a prefix for the resource group name. Note that you must use unique prefixes within your subscription.

Network settings allow using the preliminary created virtual networks in your target location and optionally creating a public IP address for build agent machines.

After the image is added and the cloud profile is saved, TeamCity does a test start for virtual agents to gather the agent configuration. Once the agents are connected, TeamCity stores their parameters to be able to correctly process build configurations-to-agents compatibility.

Feedback

You are welcome to try the plugin for TeamCity, and your feedback will be really appreciated, so please share your thoughts in the comments to this blog post, in the forum or issue tracker.

Happy building in the cloud!

Posted in Features, FYI, How-To's, Tips&Tricks | Tagged , , , , | Leave a comment

AWS CodeDeploy Runner plugin

Good news for those of you who use both TeamCity and Amazon Web Services, and those who for now stick with only one of the above, but are looking for a good integration.

With the help of the new TeamCity AWS CodeDeploy Runner plugin, your application can now be automatically deployed directly to your fleet of EC2 instances, on-premises instances or both in the scope of the CodeDeploy application.

AWS CodeDeploy is a fully managed, high-scale deployment service that can work with any kind of deployable content like code, web and configuration files, executables, etc. and with almost any kind of target platforms.

Generally, any deployment consists of four main steps:

  • preparing deployable content
  • uploading it to a repository
  • triggering a deployment
  • viewing the deployment status

TeamCity AWS CodeDeploy Runner plugin helps to automate and simplify all the mentioned tasks by adding a build step which packages the build output as a revision, uploads it to an S3 bucket and registers it in CodeDeploy. For the registered revision, it can run a deployment and optionally monitor and show its status using the TeamCity progress and error reporting.

To set up a deployment, one basically needs to make the following five steps (assuming  the initial account set-up is complete):

  • set up target instances
  • plan a revision
  • set up a CodeDeploy application
  • add an AWS CodeDeploy step to a TeamCity build

Target instances

By default each deployment can target up to 50 instances which must be running an AWS CodeDeploy agent and either be tagged or belong to an Auto Scaling group. Amazon EC2 instances must be running with an IAM instance profile attached –  for the AWS CodeDeploy agent to verify the identity of the instance – and must belong to the same region as the CodeDeploy application.

AWS CloudFormation provides a template which creates and configures new, single Amazon EC2 instances based on Amazon Linux or Windows Server with the AWS CodeDeploy agent already installed. For more information on setting up both EC2 and on-premises instances, see Configure Instances.

Application revision

An application revision is an archive containing deployable files accompanied by the application specification file appspec.yml which is a YAML-formatted file used by CodeDeploy to determine where to place the application files on the target instance, and which lifecycle event hooks to run in response to deployment lifecycle events – that’s where most of the magic happens. The archive is uploaded to an existing S3 bucket and registered in a CodeDeploy application.

The TeamCity AWS CodeDeploy Runner accepts either a ready-made application revision (must contain appspec.yml) archive or a list of paths to package. For the latter Ant-style wildcards like dir/**/*.html are supported.

CodeDeploy application

A CodeDeploy application is a place where all the deployment resources are put together: the revision location, deployment group, deployment configuration and other settings.

An AWS CodeDeploy deployment group identifies a collection of instances based on their tags, Auto Scaling group names, or both. Multiple application revisions can be deployed to the same instance, and one and the same application revision can be deployed to multiple instances.

The deployment configuration specifies a set of deployment order rules and the success and failure conditions. It specifies the number (or percentage) of instances that must remain available at any time during a deployment. One may either use one of the default configurations, or create a custom one.

AWS CodeDeploy TeamCity build step

After one has a working CodeDeploy application,  he or she only needs to type in the settings such as  region, credentials, S3 bucket parameters, CodeDeploy application and deployment group name, and run a TeamCity build.

Screen Shot 2016-03-17 at 17.32.35

Posted in Blogroll, Features, FYI, How-To's | Tagged , , , | 2 Comments

TeamCity Digest #5

teamcity-digestPlease welcome the latest digest of new publications about TeamCity usage and different aspects of CI/CD processes that it covers.

We’ll start this one with the post by Paul Stovell in the official Octopus Deploy blog about how some of the new major changes in ASP.NET Core 1 and the dotnet CLI have impacted build and deployment pipelines for developers: ASP.NET Core 1 build & deployment pipelines with TeamCity and Octopus.

This post helps understand how to automate your .NET website deployment and relieve some of the stress induced by manual deployment. The toolset of the author is quite standard for the task: Visual Studio, TeamCity, and Octopus Deploy: How to level up your .NET website deployment process.

If you are a .NET developer who likes to obfuscate code and wants to automate the job, here is an example of how it can be done with the help of TeamCity: Using Dotfuscator to obfuscate code within TeamCity.

For a growing Docker users army out there, here is a detailed 3 piece guide for implementing Continuous Integration with TeamCity and Docker by Gabriel Schenker: Part 1, Part 2, Part 3.

A recent article in the Code Magazine by Jeffrey Palermo, who delivered a TeamCity webinar in February, covers many of the concepts of Continuous Delivery as a part of modern DevOps movement, and particularly focuses on how cloud technologies can help in the process: DevOps and Continuous Delivery: Made for a Cloud World.

To add some diversity to the .NET-biased content of this blog post, here is some practical information for PHP developers. If you are interested in getting code coverage out of the PHPUnit tests and being able to see them in your builds, you might find this post useful as it goes over some very basic build steps and configs needed to get PHPUnit tests run and code coverage generated: Setting up Code Coverage on TeamCity 9.1.

If you have published an article, a blog post, a video or some other form of content on the subject of using TeamCity, or know someone who did, we’ll  appreciate if you make sure it reaches our attention: such content will be promoted to a broader audience of TeamCity users.

Access all TeamCity Digests.

Posted in Blogroll, Digest, FYI | Comments Off on TeamCity Digest #5

TeamCity Take on Build Pipelines

Although it is possible to setup  build pipelines in TeamCity, you will not find the term “build pipeline” in the TeamCity web interface or even in our documentation. The reason is that we have our own term: “build chain“, which we feel describes TeamCity’s build-pipeline-related features better.

First of all, while the word “pipeline” does not impose sequential execution, it still implies it. At the same time, in TeamCity a build chain is a DAG – directed acyclic graph, and as such a traditional build pipeline is just a special case of the TeamCity build chain. But there are other differences as well.

Source code consistency

Source code consistency implies that all builds of a build pipeline will use the same source code revision. Traditionally, when build pipelines are described, not much attention is paid to this problem. After all, if someone needs consistency, it can be achieved by sharing data between builds using a file system or by tagging the source code in the repository and then fetching the same tag in all builds.

But from our experience in the majority of cases users need consistency for all of the builds participating in the pipeline. For instance, if you build parts of your project on different machines in parallel and then combine them in the last build, you must be sure that all of the parts have used the same source code. Actually, any time when you need to combine results of execution of several parallel builds, you need to be sure all these builds have used the same source code.

You can spend time and achieve this consistency somehow via build scripts, but since many developers face the same problem, obviously this is the area where CI tools should help. TeamCity provides source code consistency for all builds in the chain automatically. It does it even if the build chain uses different repositories and even if the builds use repositories of different types – Git, HG, SVN, Perforce, TFS.

Triggering

We also thought that traditional approach to triggering – when the next build of the pipeline is triggered after the previous one is finished –  is limiting, especially in terms of optimizations that CI system are capable of. If the whole build pipeline is known from the beginning, then it is possible to reduce build time by making some optimizations. But if the builds in the pipeline are triggered one by one, then it will always take the same time to finish completely.

For instance, say we have a pipeline C -> B -> A, where C is the first build and A is the last:

cba_chain

If we trigger the whole pipeline at once, all builds (C, B and A) will be added into the queue and, if the system already has a finished build which used the same source code revision as build C will, the build in the queue can be substituted with such finished build reducing the total build time of the pipeline. This is exactly what TeamCity does for all build chains while they sit in the queue.

But if we trigger the builds one by one, i.e C is the first, then upon finishing C we start B, and so on, then there are much fewer opportunities for a CI system to optimize something. If something triggered C, the CI system must obey it , because this is how the user configured it. Even if there are finished builds identical to C (run on the same sources revision), at this point  the CI system cannot avoid running C, as upon finishing C, build B and eventually A must be triggered as well. If we decide to not run C,  we will not run the whole pipeline, which is definitely not what the user expects.

So this is the reason why we always recommend to avoid triggering builds one by one. If the build chain is fully automated (which is what we all try to achieve these days), then ideally there should be only one trigger – at the very last build of the chain (in terms of TeamCity – the top build, build A in the example above), which will trigger all the builds down the pipeline. Fortunately, TeamCity triggers are quite advanced, you can configure them to trigger the whole chain if a change was detected in a part of it. This has an important benefit as the set of triggers that you need to maintain can be drastically decreased.

According to the data collected from our own build server, due to this and other optimizations performed in the build queue, TeamCity greatly reduces the amount of work performed by agents daily:
queue_stats

Note that our server produces about 2500-3000 build hours per day, so if there were no optimizations like this, we’d have to add more agents, or our builds would be delayed significantly.

Data sharing

Another important aspect is how you pass data from build to build in a build chain. Obviously you can use artifacts for this task: the previous build produces artifacts and the next one uses them as input. But note that in many cases you cannot rely on the fact that the next build will be executed on the same machine as the previous one. Even if it does, a few other builds could be executed on this machine and they could remove the results that you wanted to share. Fortunately, publishing artifacts to TeamCity and then using artifact dependencies to retrieve them solves all these problems.

Besides,  in TeamCity you can also share parameters between builds. All builds in TeamCity publish their parameters upon finishing: system properties, environment variables, the parameters set in their build configuration at the moment of the build start, the parameters produced by the build with the help of service messages, etc. And all of them can be used in the subsequent builds of the build chain. For instance, this makes it possible to use the same build number for the whole build chain.

Find out more on sharing parameters.

Build pipelines view

The traditional approach to build pipelines implies that there is a dashboard showing all pipelines and changes which triggered them. Given that some of our customers have build chains consisting of hundreds of builds (we’ve seen chains with up to 400 builds) and in large projects there can be hundreds or even thousands of source code changes per day, it is obvious that a simple dashboard will not work.

TeamCity has the Build Chains tab on the build configuration level displaying an aggregated view of all of the chains where builds of this build configuration participated. It can surely be used as a dashboard to some extent, but with large build chains it quickly becomes unusable.

build_chains

Fortunately in TeamCity each build of the chain also shows the state of all of the builds it depends on. So by opening the build results of the last build in the chain (the top build) you can see the state of the whole chain: which builds are running, which builds failed, which artifacts were produced by each build, which tests failed for the whole chain, etc.

build_results_deps

Summary

Hopefully this article sheds some lights on to how build pipelines can be configured and used in TeamCity and why they work this way. To sum up:

  • a build pipeline in TeamCity is called a build chain
  • source code synchronization in TeamCity comes for free, you don’t need to do anything to achieve it
  • the data from build to build can be passed with the help of artifacts or parameters
  • if possible, avoid one-by-one triggering: the bigger the parts you trigger, the better
  • monitor the build chain results using the results page of the last build in the chain or using the Build Chains tab
Posted in Blogroll, Features, FYI, How-To's, Tips&Tricks | 4 Comments

TeamCity 10 (code-named Indore) EAP is open

Greetings, Everyone!

The TeamCity development team is proud to announce the Early Access Program (EAP) for TeamCity 10.0. Following the tradition of naming our version after Indian cities, this one is code-named Indore.

The preliminary date for TeamCity 10.0 is June 2016. If you want to be the first to learn about new features in 10.0, keep watching for our EAP builds: we expect to publish them more often now.

Indore 10.0 EAP1 (build 40941) main features

  • TeamCity now can integrate with GitHub, GitHub Enterprise or BitBucket Cloud: with the integration configured, setting up projects which use GitHub or BitBucket Cloud is a piece of cake.
  • Build 40491 comes with cross-platform TFS integration: to work with TFS you no longer need to install a TeamCity server on a Windows machine.
  • This EAP comes with reworked user interface for Amazon EC2 cloud integration and some other new features and improvements in both Amazon and VMware vSphere plugins.
  • Those of you who use several TeamCity severs will appreciate the new TeamCity projects pop-up: written from scratch, it provides better performance and allows browsing projects and build configurations on multiple servers.
  • If you use Perforce with streams, try out our experimental feature branches support.
  • and much more!

As usual, our EAP builds come with 60-day Enterprise evaluation license.
See our release notes for details, and, if you want to test the newest TeamCity features, download this EAP version. We recommend installing the EAP version on a trial server.

Give this version a try and share your feedback, as we always listen to what you have to say!

Help us improve the new TeamCity and happy building!

Posted in Blogroll, Bugfix, EAP, Features, News & Events | 8 Comments

TeamCity Digest #4

teamcity-digest

In this fourth TeamCity Digest we want to draw your attention to several publications of the last couple of weeks. We’ve also decided to introduce a new section in our digests on the  TeamCity plugins recently added either by our users, or by our team members.

So read on and share your thoughts with us.

Matthew Adams of endjin shares his experience of using SpecFlow 2.0, NUnit3, Visual Studio and TeamCity.

The other blog post describes how TeamCity can be used to build and deploy the code and Sitecore items to a Sitecore instance when this instance and the TeamCity build server reside on different physical servers: Continuous Integration with Gulp and Sitecore Powershell Extensions Part 2 – TeamCity Integration.

Ahmed Gadir describes a process of using TeamCity to generate release notes on every build, with the user stories stored in Jira: How can TeamCity generate Release notes from JIRA during a build.

CTO of Keluro shares their story of setting up Jekyll website on Azure using TeamCity with triggers on Git repository branch changes: Hosting Jekyll website on Azure Web app using TeamCity automated deployment.

A couple of useful tips for those who use Powershell and TeamCity: TeamCity queries with powershell (for sake of healthy develop).

Recently added plugins:

  • TeamCity Theatre – a .NET application to monitor your TeamCity builds in your browser or on a TV
  • TravCorp radiator – JavaScript build radiator – improved fork of tc-radiate
  • SwiftCity – TeamCity REST API client in Swift
  • ILMerge Meta-Runner – meta-runner to run ILMerge (utility to merge multiple .NET assemblies into a single assembly)
  • TeamCityZen – provides a command line tool which can be run as a build step to send notifications based on @user mentions in change comments

Should you write or find a piece on TeamCity, or have a plugin which is not yet listed on our plugins page, feel free to share it with us.

P.S. No, we can’t resist sharing this cool hardware TeamCity ‘Deploy’ button with you: 

Read all TeamCity Digests.

Posted in Blogroll, Digest | Comments Off on TeamCity Digest #4

Webinar recording: Implementing Continuous Delivery with TeamCity at the Helm

The recording of our February 24 webinar, Implementing Continuous Delivery with TeamCity at the Helm, is now available on the JetBrains YouTube channel.

In this webinar Jeffrey Palermo of ClearMeasure goes over how to set up a continuous delivery pipeline managed by TeamCity, very similarly to how Microsoft’s NuGet team manages NuGet.org. The same chain of tools is used, and the same techniques. You will see, in detail, how to configure TeamCity to run a fully-capable deployment pipeline complete with unit tests, integration tests, acceptance tests, staging, deployment smoke tests, and gated & approved production deployments.

Below is the precise timeline of the webinar and select Q&A.

Continue reading

Posted in Partner, Webinar | Comments Off on Webinar recording: Implementing Continuous Delivery with TeamCity at the Helm

Implementing Continuous Delivery with JetBrains TeamCity at the Helm, February 24th Webinar

Join us Wednesday, February 24th, 15:00 – 16:00 GMT (10:00 AM – 11:00 AM EST) for our free live webinar, Implementing Continuous Delivery with JetBrains TeamCity at the Helm with Jeffrey Palermo of Clear Measure.

teamcity_webinar_Implementing_Continuous_Delivery_with_JetBrains_TeamCity_at_the_Helm

TeamCity has long managed Continuous Integration workflows. And it also plays well in the DevOps world. DevOps is a term representing the convergence of development and operations. Both the development and operations communities like to own the term, so it has accumulated multiple meanings. With the Agile Manifesto, developers have jettisoned annual software releases for processes that allow releasing software multiple times per day. To accomplish this, teams use rigorous suites of automated tests to ensure that every change to the software is a solid step forward rather than an accidental step backward.

In this webinar, you will learn how to set up a continuous delivery pipeline managed by TeamCity, very similarly to how Microsoft’s NuGet team manages NuGet.org. We will use the same chain of tools, and the same techniques. You will see, in detail, how to configure TeamCity to run a fully-capable deployment pipeline complete with unit tests, integration tests, acceptance tests, staging, deployment smoke tests, and gated & approved production deployments.

Space is limited, please register now. There will be an opportunity to ask questions during the webinar.

About the presenter:

Wes McClureJeffrey Palermo is a Managing Partner and CEO of Clear Measure, an expert software engineering firm based in Austin, TX with employees across North America. Jeffrey has been recognized as a Microsoft MVP for 10 consecutive years and has spoken at national conferences such as Tech Ed, VS Live, and DevTeach. He’s the purveyor of the popular Party with Palermo events and is the author of several books, articles, and hundreds of pithy Twitter quips. A graduate of Texas A&M University, an Eagle Scout, and an Iraq war veteran, Jeffrey likes to spend time with his family of five out at the motocross track.
Posted in Partner, Webinar | 2 Comments

Live webinar: YouTrack + Upsource + TeamCity Come Together

Notice: this post is originally published in the Hub blog. However, your comments are welcome here as well.

If you are using more than one team tool from JetBrains, or planning to start,  you definitely need to  join us Wednesday, February 17th, 15:00 – 16:00 GMT (10:00 AM – 11:00 AM EST) for our free live webinar with Dmitri Nesteruk.

hub_webinar_YouTrack+Upsource+TeamCity_Come_Together_2

In this webinar, we are going to demonstrate:

  • How our teamware tools can be connected to a single entry, user and permission management point (Hub).
  • How to manage users, groups and permissions in Hub.
  • How to connect different tools to one another.
  • The benefits that the various directions of integration offer to teams using YouTrack, Upsource and TeamCity together.

Space is limited, please register now.

This webinar is geared towards developers of different proficiency regardless of programming language of choice. During the webinar there will be an opportunity to ask questions. The recording will be available after the webinar.

Posted in Uncategorized | Comments Off on Live webinar: YouTrack + Upsource + TeamCity Come Together

TeamCity 9.1.6 is out now

Greetings, everyone!

Today we’ve published another bugfix release, TeamCity 9.1.6, resolving over 50 issues.

Build 37459 mostly fixes minor problems – the full list is available in our Release Notes. We hope that a number of enhancements in our NUnit 3 support and fixes for the irksome issues with NuGet should further improve TeamCity experience for .Net users.

As with any bugfix release, with TeamCity 9.1.x versions you can freely upgrade or downgrade, so be sure to check the Upgrade Notes, and download and install the latest TeamCity build!

Stay tuned for our latest news: we’ve been putting a lot of effort into the new TeamCity version, and it won’t be long before we announce the EAP for the jubilee TeamCity 10!

Happy upgrading and building!

Posted in Bugfix, EAP, News & Events | Comments Off on TeamCity 9.1.6 is out now