Benchmarking TeamCity

How many agents can be handled by TeamCity server?

This is our official reply to the question in the title:
The latest TeamCity version is known to work well with up to 300 build agents (300 concurrently running builds actively logging build run-time data). In synthetic tests the server was functioning OK with as many as 500 agents (the server with 8 cores, 32Gb of total memory running under Linux, and MySQL server running on a separate comparable machine).

This reply is based partly on our own experience and partly on our internal benchmarks. In this post I’ll explain what kind of benchmarks we use and will also show some of the results measured against the latest TeamCity version – 9.1.1.

We’ll start with the TeamCity architecture first: this will let us understand results better.

As you already know, TeamCity consists of a server and a number of agents. Agents run builds and send results (build log, artifacts, tests, etc.) back to the server. Most of the time the results are being sent in real time, while build is still running, which allows us to show a nice real-time progress of the build in the web interface, send notifications early, and so on.

An agent sends results to the server by the RPC protocol established over HTTP. During processing of these results, the server stores data in the build log on the file system and can also write some data into the database. For instance, all statistic values, as well as the information about tests (name, duration, status) are stored in the database.

Both the agent and the server maintain buffers for the build results data: the agent sends results to the server when the buffer becomes full, the server also processes results in packs, because it greatly improves performance. To reduce concurrency costs on the server side and in the database, all build results on the server side are processed by N worker threads, where N roughly equals to the number of cores. So even if we have 300 agents sending results at the same time, they will be processed on the server with the concurrency not greater than the number of cores.

So what happens if the server cannot sustain the load? All running builds have bounded message queues, so if these queues become full because workers cannot keep up with the data from agents, agents will get blocked on an attempt to add new data from builds. As a result, the build time will start increasing. When this happens, we can observe that the build script itself has already finished on the agent, but the server still did not mark the build as finished because it is still processing the build results.

This brings us to a simple idea of the benchmark: we need to generate build logs from a lot of agents for some time with a constant rate; then we need to stop the load generator and measure the amount of build logs processed on the server up to this time. Once we know the duration of the benchmark and the rate of build log generation, we can estimate the amount of build logs which should’ve been processed. Having this delta (estimated amount – actual amount) we can judge whether the server is ok with the load or not.

Meet Benchmark plugin

This benchmark idea has been implemented as a TeamCity plugin. The plugin adds additional tab under the Administration -> Diagnostics page. On this tab you can configure benchmark parameters, such as: the number of concurrently running builds, the duration of benchmark, the build log output rate and others.


Some of the fields require additional explanation:

  • Build log output rate – the number of log lines produced by a build per second
  • Percent of successful tests in build log – assuming that one line of the build log represents a test, this is the percent of build log lines to allocate for tests reporting. So if the build log output rate is set to 30 lines per second, and the percent of successful tests is set to 50, then the load generator will produce 15 tests per second and 15 other random log lines for each build.
  • Percent of failed tests in build log – the meaning is the same as with the successful tests percent; note that the sum of and cannot exceed 100.

Note that the build log output rate and percent of successful and failed tests can be pre-calculated based on builds history of projects already configured on the TeamCity server (see “Pre-load parameters” button).

The benchmark workflow is pretty simple. When the benchmark starts, it:

  • creates a dedicated agents pool named Benchmark with the number of agents equal to the number of configured concurrently running builds
  • creates a project with build configurations where builds will start: one configuration for each build
  • adds all builds to the queue and waits till builds start
  • spawns the load generator as a separate java process and waits till it finishes
  • cleans up the data: stops builds, removes the project and build configurations, removes the agents and the agent pool; cleanup is also performed if the benchmark is interrupted using the Stop button.

The log file for each benchmark is stored in a separate file under the <TeamCity logs directory>/benchmark folder. If additional comments are specified, they will be stored in the benchmark log as well.

Benchmark results

So we’ve done some measurements on our own hardware:
TeamCity server:

  • CPU: Core i7-4790, 3.6GHz
  • RAM: 32Gb
  • SSD: Intel 530, 240Gb
  • HDD: Seagate Barracuda ES (ST3500320NS), 7200RPM

Database server:

  • CPU: Core i7-4790, 3.6GHz
  • RAM: 32Gb
  • SSD: Intel 530, 240Gb
  • HDD1: Western Digital Velociraptor (WD3000HLFS), 10000RPM
  • HDD2: Western Digital Caviar Blue (WD6400AAKS), 7200RPM

Ubuntu 14.0.4 was installed on both servers. In case of the TeamCity server, the TeamCity data directory was placed on an HDD. On the database server, the data files and transaction logs were placed on an SSD. In all of the cases the Ext4 file system was used.

Percona server 5.6.25 was installed on the database server machine. We’ve selected Percona because it is well known for it’s performance optimizations compared to regular MySQL. The following parameters were changed in my.cnf file:

thread_handling=pool-of-threads # Percona and MariaDB only

TeamCity server was started under Java 1.8.0_45 with Xmx set to 3Gb.

To make the benchmark results more realistic, we set up a few open source projects and measured their build log output rate using the “Pre-load parameters” feature. See the results below:

Project name Build log output rate per build Percent of build log used by successful tests Resulting tests rate per build
commons-math 71 lines/sec 80% 57 tests/sec
JUnit 46 lines/sec 61% 28 tests/sec
Maven 45 lines/sec 28% 12.6 tests/sec
Spring Framework 20 lines/sec 50% 10 tests/sec
Netty 30 lines/sec 21% 6.3 tests/sec
Hibernate ORM 60 lines/sec 5% 3 tests/sec

Let’s see how these numbers are calculated:

  • build log output rate = <total number of log lines> / <build duration>
  • tests rate – assuming that each test is represented in the build log by at least one line:
    tests rate = <build log output rate> * <number of tests> / <total number of log lines>

As you can see, these formulas do not take the test duration into account. So even if there are lots of fast tests with the duration of 1ms, but besides running tests a build also performs other time consuming tasks, the resulting tests rate will be far from 1000 per second. This is especially true when many builds are running concurrently on the real server. In this case the builds will be in different states: some can run tests, others download artifacts, compile code, etc, etc. So we can safely say that it is unlikely that there will be a situation when all of them report tests with the rate of 1000 per second at the same time. As a result, the real load which can be sustained by the server is much bigger than one would expect by simply looking at the duration of individual tests.

Final results

For each of these projects we measured the maximum number of concurrently running builds our system can sustain without significant increase of the build duration (10% increase in build duration was considered acceptable). See the results below:

Project name Max number of concurrently running builds CPU usage (server/database)
commons-math 320 42% / 20%
JUnit 570 40% / 18%
Maven 990 50% / 15%
Spring Framework 1300 52% / 15%
Netty 1600 58% / 10%
Hibernate ORM 1650 60% / 9%

Some observations:

  • even with this hardware which is not quite optimal for the server tasks, some very promising results can be achieved
  • the more tests produced by the build per second, the smaller the resulting number of agents concurrently running builds; in our case commons-math has rate of 57 tests per second which may not sound like a lot, but surprisingly the majority of projects have much lower rate of tests. In general, the bigger the project, the slower its tests, which is understandable, as besides tests, build scripts also perform lots of other time-consuming tasks
  • the bigger the rate of the build log output, the bigger the load produced on the TeamCity server machine; with the large number of agents, the CPU usage on the TeamCity server machine reached 60%; obviously, it is not a good idea to dedicate 60% of CPU just to process data from agents as there are other CPU intensive tasks which should be done by TeamCity server
  • the load generator worked as a separate java process, and at peak times it took some CPU as well, although this time was negligible comparing to the CPU time used by the server
  • the CPU usage on the database server only reached 20% which probably can be explained by a very simple workload imposed by our test – mostly inserts, no complex queries at all

Note that the results of this benchmark cannot be applied blindly simply because this benchmark is a synthetic test and, like many other synthetic tests, it does not entirely represent the real life. For now it only measures one aspect of the server performance. It does not cover things like artifacts downloading and publishing, user requests, VCS operations, etc, etc.

On the other hand, if you plan to move your server to new hardware, this benchmark can help you to tune the hardware, disks and database parameters to achieve better performance.

How to run benchmark on your own server

The Benchmark plugin is available for download.

The plugin is compatible with TeamCity 9.1.x.

If you want to try this plugin on your hardware, please read our suggestions below.

The plugin creates agents and each agent needs to be authorized, which means you should have enough agents licenses to run benchmarks. Besides, using this plugin on the production server may be a tempting idea, but the plugin can impose a significant load on the system, making it hard to use by other users. Moreover, in case of bugs in the plugin or server crashes, you can end up with lots of data which was not properly cleaned up – agents pool, agents, projects, etc. Finally, it may not be easy to fine tune the production system for better results as it requires downtime.

Because of that we recommend to setup a separate server, obtain an evaluation license from our site to be able to run the server with unlimited agents, and then run benchmarks on this system. Note that using the latest TeamCity version is also recommended as it may have significant performance improvements comparing to the version you’re using now.

Take a look at our documentation if you’re going to use MySQL or PostgreSQL database.

We would not recommend using HSQLDB for this benchmark: since we don’t recommend this database for production use, benchmarking against it is pointless. It’s a good database engine, but it is not as reliable and scalable as “big” databases. Big databases implement sophisticated approaches to keep your data safe (see for example: or HSQLDB being a database written in Java does not have a chance to implement similar reliability as it does not have proper access to hardware.

While running benchmarks, make sure to run them a few times, or at least run them with the duration not less than 180 seconds as this makes results more predictable. It is also recommended to perform a few warm up benchmarks after the server startup. Finally, if you choose to tune some parameters in the database or elsewhere, make sure to change one parameter at a time. Otherwise you may fail to notice which parameters give you better results.
Benchmarking is fun, but be prepared to spend tons of time on it!

We hope that you will use this plugin to assess your server performance, so if you do decide to run this benchmark, we will be really interested in seeing your results.

Happy benchmarking and building!

Posted in Design decisions, Features, FYI, How-To's, News & Events, Uncategorized | Leave a comment

TeamCity 9.1.1 to the rescue!

Greetings, everyone!

Since TeamCity 9.1 was released, several issues requiring a quick fix on our side have been reported to our tracker.
The most noticeable of them are:
- problems with building .NET 4.6 projects and proper detection of .NET 4.6 on agents
- problems with running NUnit tests on Linux
- Swabra exclude rules stopped working in some cases
- problems with the TeamCity server launch on some platforms

In 9.1.1 all these problems are fixed.

We also added one feature to Amazon EC2 integration which had already been cooked for quite some time:
- TW-34196 – ability to set IAM role on a running instance

Overall TeamCity 9.1.1 addresses more than 60 different problems. See the complete list of fixed issues in our release notes.

If you’re already using TeamCity 9.1, make sure to install this update. If you have not upgraded yet, please consider doing so, as all the new fixes are to be applied to the 9.1.x branch; no updates for TeamCity 9.0.x are planned.

Download and install TeamCity 9.1.1! As with any other TeamCity bugfix update, this version does not introduce changes in data format and can be easily reverted to TeamCity 9.1.

Happy building!

Posted in Bugfix, EAP, Features, Uncategorized | 2 Comments

TeamCity 9.1 release: truly historical and very personal builds

Greetings everyone!

So here it comes – after half a year and over 500 fixes since version 9.0 was released, despite being a minor version, TeamCity 9.1 brings you major improvements!

The new build is coming out with significantly enhanced capabilities to store the project settings in the VCS: now the settings in your build can be different from those currently on the TeamCity server, which means that you can:

  • run true historical builds: use a particular revision of the source code and run a build with the corresponding project settings
  • store different settings in different branches, and run a build in a particular branch with the settings from this branch
  • change project settings in your IDE, and run a personal build/pre-tested commit with these custom settings.

Good news for .Net developers – this release brings enhanced .NET tools support:

  • anticipating the upcoming release of Visual Studio 2015, TeamCity already supports the latest versions of the tools in the Visual Studio 2015 lineup
  • Windows PowerShell 5.0 is also supported
  • the new Visual Studio Tests runner integrates the MSTest and VSTest runners
  • Mutual support between TeamCity and NUnit 3.0 guarantees compatibility of NUnit and TeamCity future releases.

As with every release, we were busy with UI improvements, and one of the most noticeable is probably ANSI-coloring support in our build log, which presents endless possibilities for those with artistic aptitude: buildLogArty

For the full list of new features and fixes see our What’s New on the product web site and our documentation.

Download and install TeamCity 9.1 now!

Happy building!

Posted in Bugfix, EAP, Features, FYI, News & Events | 6 Comments

TeamCity 9.1 RC is out today

Greetings, everyone!

We are happy to announce the TeamCity 9.1 release candidate! We’ve been putting a concentrated effort into finalizing the new features and fixing issues, so here is our latest news.

Now unidirectional communication between the TeamCity agents and the server is enabled by default. Should you run into any issues, please let us know.

Besides, TeamCity 9.1 RC introduces a number of fixes in the Nunit 3.0 support. We’ve looked at the statistics related to the usage of code coverage tools with NUnit and decided that JetBrains dotCover will be the only coverage tool for NUnit 3.0 for now.

Other improvements include the bundled SBT plugin. See the full list of fixes in our Release Notes.

As usual, make sure to install this build in a test environment since it changes the data format, and downgrade is not supported.

Download TeamCity 9.1 RC build 36932 and let us know what you think in our forum or issue tracker!

TeamCity 9.1 will be out in a few days, so stay tuned!

Happy building!

Posted in Bugfix, EAP, Features, FYI, News & Events | Comments Off

TeamCity 9.1.EAP3 is available

Greetings, everybody!

Hot summer news from TeamCity: 9.1 EAP3 (build 36802) is available for you to download and try!

As the release time is approaching, we are focusing on polishing the new features and fixing known problems, so altogether this build contains over 90 fixes and improvements.

By default, the communication between the TeamCity server and agent is bidirectional, which causes issues with the agent firewall, now you can configure the agent to use unidirectional connection to the server! More details are available in our Release Notes.

For .Net developers, this EAP brings you a new test runner, Visual Studio Tests, which integrates the bundled MSTest runner and the VSTest console runner formerly provided as an external plugin. Now one runner supports both testing frameworks enabling TeamCity to execute tests and automatically import their test results.
Note that after upgrade to TeamCity 9.1, MSTest build steps are automatically converted to the Visual Studio Tests runner steps, while the VSTest steps will remain unchanged.

Take a look at our Release Notes for a more detailed description of features and enhancements, download and install this EAP build! As usual, your feedback in our forum or issue tracker is most welcome!

Remember that EAP build is work-in-progress and is not intended for production, so installing it on a trial server is the way to go: the new version changes the data format and TeamCity does not support downgrade.

Stay tuned for more news on TeamCity 9.1!

Happy building!

Posted in Bugfix, EAP, Features, News & Events | Comments Off

2015 TeamCity Plugin Contest Winners Announced

Greetings to all of you!

As you remember, in the beginning of April we launched TeamCity plugin contest, the first TeamCity competition aimed at creating innovative plugins to enhance the available features of TeamCity.

Since June 15th 2015, which was the deadline for submissions, our jury has been busy selecting the three best plugins and today we are happy to announce the winners!

First prize
$2,000 Amazon gift voucher and $5,000 in JetBrains store credit
Winner: Alexey Moskvin
Plugin: TeamCity password reset plugin
Description: This plugin adds a ‘password reset’ link to login screen and allows you to reset password without disturbing The TeamCity Administrator.

Second prize
$1,500 in Amazon gift voucher and $3,000 in JetBrains store credit
WinnerEvgeny Goldin
Plugin: TeamCity Webhooks plugin
Description: A plugin POST-ing JSON payload upon build finish to a number of registered HTTP listeners. Each payload contains build name, number, status, Git details (sha/repo/branch) and possibly artifacts URLs, downloadable from TeamCity and/or AWS S3 (if set up this way).

Third prize
$1,000 in Amazon gift voucher and $1,000 in JetBrains store credit
WinnerIgal Tabachnik
PluginTeamCity Visual Studio Gallery (vsix) Plugin
Description: A plugin for TeamCity that turns it into a Visual Studio Extensions (vsix) Gallery.

Special prize from TeamCity developers team
a Trip to JetBrains office in St.Petersburg or Munich
WinnerGrigory Chernyshev
Plugin: TeamCity web-ssh
Description: Plugin for connecting to ssh via web ssh directly from Teamcity.

Our congratulations to the winners! We’ll be contacting your shortly after this announcement, ping us in the comments to this post if we don’t!

We are thankful to all the participants of the competition: all together we received 31 plugin and we are currently updating our documentation to list them all. Out of 31, 23 were considered valid and we were happy to grant each of the authors a personal licence to one of the JetBrains IDE.

Here is the list of these plugins:
EnvInject Plugin
Chuck Norris Teamcity Plugin
TeamCity password reset plugin
Web Ssh
xUnit test runner
Unified Push Notifier
Server Configurations
TeamCity One-time Passwords Plugin
Teamciy Yammer Notifier
SourceForge Integration
Ansible Runner
Parameter Finder
TeamCity TargetProcess Issue Tracker plugin
TFS WorkItem Plugin

Congratulations and happy building!

Posted in Contest, Features, FYI, News & Events | 3 Comments

TeamCity 9.0.5 is out

Greetings, everyone!

Today we are announcing the fifth and most likely the last bugfix release for TeamCity 9.0, addressing about 50 different issues.

We solved some VCS-related problems, and most importantly, delivered a number of security fixes and introduced 10+ performance improvements in this version, so upgrade is highly recommended.

Read our Release notes for the full list of fixes, download the latest version and remember that upgrade to TeamCity 9.0.5 is trouble-free: given that this version uses the same data structure as the other 9.0.x builds, downgrade is possible at any time.

The next EAP for TeamCity 9.1 is on the way, so keep following us for the latest news!

Happy building!

Posted in Bugfix, EAP, Features, News & Events, Uncategorized | Comments Off

TeamCity 9.1 EAP2 is here

Greetings, everyone!
As we approach the final steps towards TeamCity 9.1 release, the excitement is building up and today we’re making available TeamCity 9.1. EAP2 which showcases a few more of the new features we’ve been working on.

This build includes over 100 fixes and improvements, and we hope you’ll find them handy.

First of all, just like the previous EAP, this build introduces several improvements to the versioned settings feature. If you already use this functionality in your project, you’ll appreciate the new interesting possibilities – see details in our Release notes .

If a tool you are using in a build has a color output mode, you’ll like the fact that you can take advantage of it now, because starting from this EAP TeamCity build logs support ANSI coloring!

We’ve got good news for .Net developers:
- TeamCity now supports Powershell 5.
- We used to support NUnit, and now NUnit supports us! Our team members and NUnit developers collaborated on several points and now NUnit detects if it is run by a TeamCity build agent and automatically switches to “TeamCity-style” reporting, using TeamCity service messages.

This build also addresses a common problem with artifact storage becoming too large: now you can store artifacts in multiple locations and specify the paths to them on the Administration | Global Settings page. The first path in the list is used to publish new artifacts, the other paths are searched when TeamCity is looking for artifacts.

We also fixed a number of performance and usability issues – for the full list see our Release Notes.

Download and install this EAP build and let us know what you think: we are working hard to get TeamCity 9.1. released soon and your feedback in our forum or in the issue tracker is as valuable as ever!

Like any EAP, this build is not intended for production, so make sure you install it on a trial server, because it changes the data format.

We’ll be releasing TeamCity 9.1 soon, so stay tuned!

Happy building!

Posted in Bugfix, EAP, Features, FYI, News & Events | Comments Off

TeamCity Plugin Contest 2015: submissions are closed

Dear readers!

The point of this post is to heartily thank every one of the participants who spread the word, joined our TeamCity Plugin Contest and spent their valuable time to help others get the best of TeamCity!

Of 156 people registered for the contest 21 submitted valid plugins, and we were happy to grant them a personal license for one of the JetBrains IDEs: all together our participants received 9 IntelliJ IDEA, 4 ReSharper, 4 PyCharm, 2 WebStorm, and 2 dotTrace licenses simply for submitting a valid plugin. Congratulations!

Submissions for the contest are closed, now begins the judging.

Our jury is in the process of selecting the three lucky winners, who will be announced on June 30, 2015, and nominees will be contacted.

So keep your hopes up, be patient and follow our blog or twitter for the winners announcement.

Thanks again and happy building!

Posted in Blogroll, Contest, News & Events | Comments Off

How TeamCity Shines for You, Webinar Recording

The recording of our May 26th webinar with Wes Higbee, How TeamCity Shines for You, is available on JetBrains YouTube channel.

In this webinar Wes goes over some of the outstanding features of TeamCity which make development and delivery of software a pleasure. The webinar was divided into three main themes:

  • Testing and Visual Studio IDE Integration
  • IntelliJ TeamCity IDE plugin
  • Pipelines / Build Chains

Below is the precise timeline of the webinar and select Q&A.

Continue reading

Posted in Blogroll, Features, Webinar | Comments Off