How TeamCity uses TeamCity
Overview
TeamCity was started back in 2005 out of our own need to have a flexible and powerful continuous integration server that could deliver functionality beyond the offerings that were available at the time. Since then, much of the functionality in TeamCity has been as much a result of our own requirements as that of our users, as we continue to use it throughout JetBrains daily.
While it probably doesn’t come as a surprise to learn that we use TeamCity ourselves, in this post we’ll describe a few details on how the team uses it internally and how it’s evolved based on own needs.
Daily Releases
You probably might know that TeamCity usually does one major and one minor release per year. Each one of these is usually preceded with an EAP program and several bugfix releases afterwards, which pretty much results in a release every other month.
Things are completely different inside JetBrains however – we continuously deliver TeamCity every working day. Here’s how it works:
-
Early afternoons, usually around 4pm, while most teams are still working, we start a new nightly build that will compile a new TeamCity package.
-
While the build is running, two developers from the team (Duty Devs) review all commits made by the team during the day and if they both agree commits won’t break critical parts of the application, such as areas responsible for running and scheduling builds and build agent auto-upgrade, they put a special tag on the build marking it as “safe for deployment”.
-
After the build is complete and package is ready, TeamCity runs a series of integration tests (a bit more on that below) against the newly compiled package. If these tests fail, the package won’t be released.
-
If all of the previous steps are successful, then at 6 am – time with minimal development activity at JetBrains – TeamCity starts a self auto-upgrade. Why 6 am and not earlier? Primarily because some people at JetBrains start work later in the afternoon/evening and stay well into the earlier hours. Usually though nobody is at work at 6 am!
-
Since all builds are run on agents and these agents operate even if the server has downtime, we have a TeamCity agent installed on the same machine as the server and it’s this agent that performs the upgrade procedure.
Daily releases not only provide us with immediate feedback on the features we use, but also allow us to extensively test the stability of the product before making any public releases.
TeamCity Configurations
Version Control
Our main version control systems in TeamCity are Perforce and Subversion. Why not Git or Mercurial you may ask. Primarily because they’re already being used extensively inside JetBrains, so we need to dog-food those that aren’t. That’s not to say we don’t use Git or Mercurial, as some parts of TeamCity also use these. In fact we have TFS too! This also provides us with a good scenario for multiple VCS root projects.
Build Tools
TeamCity is written in Java and you might wonder which build tool we use. Actually all. We use Ant, Maven, Gradle and IntelliJ IDEA build system. In fact you can build IntelliJ IDEA artifacts and run IntelliJ IDEA configurations from TeamCity exactly as you would do inside IntelliJ IDEA.
Build Chains
Build chains is TeamCity’s way to configure build pipelines, which allows amongst other things, to optimize build times by creating build configurations and defining dependencies between these. The build chain is actually a Directed Acyclic Graph (or DAG), so a certain build is started only when all of its dependencies are built and it can use the results of these upstream.
Dependencies can be snapshot, meaning all originate from the same source commit, or artifact, i.e. a result from the output of a build. On the TeamCity team we use build chains extensively, and our entire build chain is built against sources, that is, a snapshot dependency, even if these come from different version controls (which they do, and more on that later).
Consequently, the Build Overview Page of the highest build in the build chain is both a Dashboard and a Total Summary report. From here we can see the build results for the entire chain and also see where and why a certain build has failed.
Build Optimization
As the TeamCity project grew, as well as the company, it started to become more apparent that teams needed faster turnaround when it came to build results, and build chains wasn’t enough. That’s when we started paying more attention to not run builds unless strictly and absolutely necessary.
The first step was Build Chain Optimizations – when TeamCity queues a build chain, it checks whether all builds in the queue have any changes and does not run a build against a configuration without any changes.
The build queue is also constantly optimized so that TeamCity takes into account upcoming builds. As such if a build is queued and a new commit triggers another build, if this latter one includes changes of the current one queued, then the current one would be dropped in favor of the new one.
Both of these optimizations are important in keeping the queue small and build times low.
We then took it a step further with incremental builds – builds that are run only against a certain subset of changes. For example, if a certain module and its dependencies are not changed, it won’t be built nor will the tests run. This allowed us for instance to reduce the average time necessary to build and test TeamCity significantly.
While it greatly helps developers to decide whether it’s safe or not to commit/merge the change, we still run all tests for most build configurations just in case.
Cloud Integration
At a certain point we realized that typical way to create and manage build agents weren’t satisfying our needs so we started moving aggressively towards scaling build agent farms using cloud providers. The first one was Amazon EC2, followed by Microsoft Azure and VMWare vCenter. The last one now provides around 30-40% of all our build agents. Currently we have 240 build agents, and approximately 90 of these are using VMWare vCenter.
As we started to actively use the vCenter plugin, many new features came up – such as
- Smart clone: clone VM only if necessary and do not delete after a stop.*
- Automatic update: when configured VM snapshot changes in configuration, TeamCity deletes previously cloned VMs and start new ones.
As a consequence of the above features, updating JDK on 15 build agents is 15 times faster and 100 times less irritating than before.
*Smart clone is currently only available internally and it’s something we’re actively experimenting with.
Testing Features
TeamCity supports lots of different technologies and some of them are platform-specific, such as .NET technologies (MSBuild, NUnit, FxCop, etc.), Xcode, Powershell, Unix shell scripts, etc.
Many of these are used inside JetBrains by different teams, and given that we release TeamCity daily, it’s really important that these are stable and there are no failures. That’s why we have a bunch of Integration Test configurations that cover the majority of functionality provided by TeamCity server and agents.
Operating systems that we test against include: Windows, Linux, Mac OSX, FreeBSD and Solaris. When it comes to database we also test all these ones we support which include MSSQL, Oracle, PostgreSQL and mySQL, as well as the internal one of course (HSQLDB).
In total we have over 40 different configurations in our TeamCity Integration Tests project that test different areas of TeamCity in different environments. Two TeamCity features that help with this are:
- Agent Requirements, allowing to queue builds only on those agents that have the necessary software installed.
- Build Configuration Templates, permitting us to reuse common settings and only configure aspects that are different per configuration.
When tests fail on certain environments that we don’t have locally on our machine (think OS Solaris, DB Oracle, IBM JDK), we use our Remote Debug feature that allows remote debugging of a certain test on a remote agent. It allows us to apply changes and restart tests.
A day in the life of a TeamCity developer
While most on the TeamCity team have their specific area of responsibility and role (i.e. Developer, Product Manager, QA, Support Engineer), there is a rotation scheme to have each team member provide support and process customer feedback.
TeamCity is a mission critical tool for many customers and organizations and the team understands that providing help to customers often cannot rely alone on support engineers. As such, developers also proactively help support engineers and customers troubleshoot issues with TeamCity aligned with their area of responsibility. This is also great motivation to make sure the code and logging you provide is good, cause when it comes to solving a problem in production, it can save you a lot of time!
Conclusion
This is just a small outline of some of the features that the team uses and how they came about. The important takeaway from this though is that “eating your own dog food” is really taken to the next level with TeamCity.
Sergey Pak, TeamCity Team
