TeamCity 2019.1.5 is available

The new bugfix release for TeamCity 2019.1 is ready to download.

We recommend you upgrade to 2019.1.5 to get fixes for several security problems and other important issues, including:

  • the problem with the incomplete publication of NuGet packages to the NuGet feed
  • unavailability of the “Default credential provider chain” option in the Amazon ECR connection settings

The full list of resolved problems is available in our release notes.

Like any bugfix update, 2019.1.5 has the same data format as all 2019.1.x versions, which allows you to easily upgrade/downgrade within these builds.

Remember to check our Upgrade Notes and use one of these options to upgrade:

Your feedback is welcome in our forum or tracker.

Happy building!

Posted in Bugfix, Release | Leave a comment

TeamCity 2019.2 EAP3 is here

Today we are presenting the final Early Access Program build of TeamCity 2019.2.

Take a first look of the following features:

  • Redesigned Build Details page
    buildpage
    Switch to the experimental UI and try our new approach to displaying build information: a build timeline, structured build log, chart of dependencies, and more.
  • Personal builds based on unified diffs
    To run a personal build without committing changes, upload a patch with changes in a unified diff format directly to the server, via the TeamCity UI or REST API.
  • Pipeline DSL
    Try an alternative approach to describing build chains via Kotlin DSL.
  • Branch filter for artifact dependency
    Set a branch filter so the build uses artifacts from the latest build with the specified tag only from a matching branch (or all branches).

The full list of over 80 new features and bug fixes is available in our release notes.

Remember that the new release changes the TeamCity data format and downgrading to the previous production version is not supported. We recommend installing this EAP version on a trial server. If you installed the previous TeamCity EAP version, you can easily upgrade with our automatic update. In other cases, you can download the EAP build or pull the Docker image with the eap tag.

As with any EAP version, this one comes with a 60-day Enterprise evaluation license for an unlimited number of agents and build configurations.

We encourage you to try the new features and share your feedback with us, on our forum or issue tracker.

Happy building!

Posted in EAP, Features | 2 Comments

TeamCity 2019.1.4 is released

Today we are presenting 2019.1.4 update, which resolves over 40 issues in the recent version of TeamCity.

Among other improvements, it brings performance optimizations in the Pull Requests plugin and few important fixes in the .NET Core support. The full list of fixed issues is available in our release notes.

We highly recommend you install this update since it addresses several security problems.

Like any bugfix update, 2019.1.4 has the same data format as all 2019.1.x versions, allowing you to easily upgrade/downgrade within these builds.

Remember to check our Upgrade Notes and use one of these options to upgrade:

Your feedback is welcome in our forum or tracker.

Happy building!

Posted in Bugfix, Release | Leave a comment

Stop Bad Code from Being Promoted in TeamCity Build Chains, with OverOps

This guest blog post is brought to you by Alex Zhitnitsky, Director of Product Marketing at OverOps.

It’s no secret that 100% test code coverage ≠ 100% of critical issues identified. Developers, build engineers, and managers are throwing everything they have at this problem – rigorous code reviews, static analysis, and every possible type of automated test they can think of, but the facts of the matter stay the same – code still breaks.

The reason for this is that testing requires foresight. You can’t predict the future with absolute certainty, and the level of foresight that’s required to prevent code from breaking in production is close to impossible. It’s a fact of life that we’ve grown to accept, and even the best engineering teams out there can’t avoid it.

Every engineer worries that the new code they deployed is going to cause issues in production, but there’s only so much they can test.

So what can be done to address this gap?

Today, most teams approach code quality only through the lens of pass/fail test results and static analysis. The complexity of what’s happening underneath the surface – the exceptions, HTTP errors, logged errors, and warnings – is not addressed, letting preventable issues slip into production.

The ability to optimize all of this operational data and serve it to developers in a form they can easily consume is what sets OverOps apart and exactly where our integration with TeamCity fits in.

Introducing runtime code quality gates for TeamCity

At OverOps, we’ve created a new way to ensure code changes don’t impact customer experience in Java and .NET based applications. By learning and mapping code at runtime, we optimize live application data and make it accessible to the entire team. This enables our users to identify code-level anomalies, prevent them from being promoted, and resolve them with the required context. Fully automated, and with no reliance on logs.

Using OverOps with TeamCity Build Chains introduces a brand new way to avoid deploying bad code to production, which takes into account caught and uncaught exceptions (including those not logged), and events like logged errors, warnings, and HTTP errors.

Normally, analyzing all these events would be extremely difficult. There’s just so much noise out there that it’s close to impossible to find the issues that matter.

This is where our machine learning algorithms kick in.

OverOps is not only aware of these events, but also deduplicates them and classifies anomalies in real-time, selectively capturing the required context to understand what happened.

The way this data meets you is through 6 new levels of runtime code quality gates:

  1. New Errors – did the release introduce any errors that hadn’t happened before?
  2. Critical Exceptions – did the release introduce any severe/showstopping errors?
  3. Increasing Error Rates – did the rate of a known error increase dramatically in this release?
  4. Resurfaced Issues – did an error that was previously resolved appear again in the current release?
  5. Total Errors Volume – did this release introduce a dangerous spike in the total volume of errors?
  6. Unique Errors Volume – did the release introduce an unusually high number of many unique (i.e. discrete) errors?

How to integrate OverOps with TeamCity

To get started with our first runtime code quality analysis, follow these 3 steps:

  1. First, let’s install the plugin. If you’re new to OverOps, check us out on overops.com by signing up and connecting your application. Once your environment is set up, you’ll be ready to get started by installing the Runtime Quality Report plugin with one click from the TeamCity UI.
  2. Now that your plugin is installed, it’s time to connect it with OverOps. Add the OverOps Runner build step from the Runner type drop down in your build configurations:
    1
  3. Fill in your OverOps environment ID and API token (accessible via OverOps), and test the connection:
  4. 2

All right! Now we’re ready to get started. You can now also name your application and deployment in the same configuration screen, and review some of the more fine-grained options. These will come in handy when you’ll want to customize how each runtime quality gate is defined and what conditions will make it block your build.

The complete documentation is available on our GitHub page.

Now that we’re set up, let’s walk through an example

Your team just ran its automated test suite on a new build, and looks like everything passed. Normally, the build would be considered stable and moved to the next step in your pipeline. However, OverOps will mark a build as unstable if it identifies code-level issues that triggered the runtime code quality gates you’ve set up:

3

We can see that in this case, build v4.41.0 was marked as unstable because OverOps identified 24 new errors that didn’t exist in the previous version, and 1 resurfaced error that was previously resolved. Now, the team will be alerted to review the new errors that were identified.

We can see each error’s rate, which represents the number of times it happened out of the overall number of times the method it’s in has been called. This is a great way to have an even more fine-grained ability to focus on the most critical issues.

But that’s not all.

To quickly triage what’s going on, clicking into each error leads you to the data you need to resolve it. This includes the complete source code and variable state for the issue at hand, the state of the environment, and more, taking into account the 7 key components of true root cause analysis.

4

For each error, you get the following types of data in this error analysis screen:

  • Type and location in the code, server, application, and deployment.
  • Times and error rate, including a time-series chart of occurrences since it was first seen.
  • Full stack trace and source code with complete variable state.
  • The last 250 logging statements, including DEBUG and INFO, regardless of logging verbosity.
  • The state of the environment, including threads, memory, and CPU stats.

In this case, we can see that a NullPointerException occurred because customer is null. Using this analysis, we can trace back the steps of how we got there and understand what needs to be done to stop it from happening again.

Does this look interesting? Then check out this 3-minute overview of OverOps to learn more, or request a demo with one of our engineers.

Posted in Guest post | Tagged | Leave a comment

Build Chains: TeamCity’s Blend of Pipelines. Part 2 – Running Builds in Parallel

In the previous blog post, we learned about snapshot dependencies and how they can be applied to create build chains in TeamCity. In this blog post, we describe how snapshot dependencies enable parallel builds.

More snapshot dependencies

Previously, we started creating the build chain for the demo application. We created two build configurations: one builds the application and the other builds the Docker image. What about tests?

Suppose there are a lot of tests and running those tests sequentially takes a lot of time. It would be nice to execute the groups of tests in parallel. We can create two build configurations, Test1 and Test2, which will execute the different groups of the tests. Both Test1 and Test2 have a snapshot dependency on TodoImage build configuration.

12-tc-pipelines-todo-tests

Since Test1 and Test2 do not depend on each other, TeamCity can execute these build configurations in parallel if there are build agents available.

Now the new aspect of snapshot dependencies is revealed. Using the snapshot dependencies, we created a build chain which is actually a DAG – Directed Acyclic Graph. The independent branches of the graph can be processed in parallel given that there are enough processing resources, i.e. build agents.

This leads us to the next question: how can we trigger such a build chain? Previously, we added the trigger to TodoImage as it was the last build configuration in the chain. Now there are two build configurations. Should we add two triggers – one to Test1 and the other to Test2? While it is certainly an option, there is a more idiomatic way to do that – with the Composite type of build configuration.

Composite build configuration

The purpose of the composite build configuration is to aggregate results from several other builds combined by snapshot dependencies, and present them in a single place. One very interesting property of this kind of build configuration is that it does not occupy a build agent during the execution.

In our example, we can create a composite build configuration that depends on Test1 and Test2 via a snapshot dependency. The new configuration will be the last one in the build chain. Now it’s possible to add a VCS trigger to that build configuration. As a result, there will be just one VCS trigger for the whole build chain.

14-tc-pipelines-concurrent-tests-annotated

Notice on the screenshot above that the two build configurations, Test1 and Test2, are running in parallel. The TestReport build configuration is also running, but it doesn’t occupy the build agent and will be marked as finished as soon as all the builds complete.

The nice feature of the composite build configuration is that it also aggregates the test results from the dependencies. In our example, if we navigate to the Tests tab of the TestReport build configuration, we will observe the list of tests that were executed in all the previous build configurations that belong to the same build chain.

14-tc-pipelines-concurrent-tests-composite

Summary

At first sight, snapshot dependency looks like a simple concept. However, it enables a lot of features in TeamCity. In the previous blog post, we saw how we can re-use the results of the builds to save build time and resources. In this blog post, we have learned that snapshot dependencies also enable better use of resources when builds are executed in parallel. Next, we will learn about building more complex projects that span multiple source control repositories, still applying the simple yet powerful concept of snapshot dependencies.

If you are interested in trying out the demo project on your local TeamCity instance, we have uploaded the configuration to a GitHub repository. The .teamcity/ directory contains Kotlin DSL settings for the build chain that we have described in this blog post. To import these settings, create a project from repository URL. TeamCity will detect .teamcity/ directory in the repository and will suggest creating the project using these settings.

Posted in Features, How-To's | Tagged , , | Leave a comment

Build Chains: TeamCity’s Blend of Pipelines. Part 1 – Getting Started

In TeamCity, when we need to build something, we create a build configuration. A build configuration consists of the build steps and is executed in one run on the build agent. You can define as many build steps as you like in one build configuration. However, if the number of steps grows too large, it makes sense to examine what the build configuration is doing – maybe it does too many things!

We can split the steps into multiple build configurations and use TeamCity snapshot dependencies to link the configurations into a build chain. The way TeamCity works with build chains enables quite a few interesting features, including parallel execution of the builds, re-using the build results, and synchronization for multiple source control repositories. But most of all, it makes the overall maintenance significantly easier.

In this blog post, we will explain how to create a build chain in TeamCity by configuring the snapshot dependencies for the build configurations.

The minimal build chain

To create a simple build chain, it is enough to create two build configurations and configure a snapshot dependency from one configuration to another.

For this example, we are using a GitHub repository. The repository includes a Gradle project that builds a Spring Boot application. In addition, there is a Dockerfile for building a Docker image.

We are going to build this project in two steps. First, we will build the application, and then build the Docker image with the binary produced in the first step.

01-tc-pipelines-todo-backend

First the build configuration, TodoApp, builds the Spring Boot application and publishes an artifact as a result. The second build configuration, TodoImage, depends on the TodoApp build configuration and builds a Docker image.

02-tc-pipelines-todoapp-config

There are two kinds of dependencies that we will need to configure: the snapshot dependency and the artifact dependency.

03-tc-pipelines-todoimg-dependencies

Artifact dependency

The TodoApp build configuration publishes an artifact to the todo.jar file. Any other build configuration can download the file by configuring the corresponding artifact dependency.

On the Dependencies tab, click the “Add new artifact dependency” button. In the dialog, locate the build configuration from which the files should be downloaded and specify the patterns to match the file path(s).

05-tc-pipelines-todo-backend-artifact-dep

The documentation page describes how to configure the artifact rules in more detail.

Snapshot dependency

To configure a snapshot dependency, go to the Dependencies tab in the build configuration settings and click the “Add new snapshot dependency” button. In the dialog, locate the build configuration it will depend on and click Save.

04-tc-pipelines-todoimg-snapshot-dep

There are a number of settings associated with the snapshot dependency. You can read about the various settings in the documentation. For our current example, the default configuration is sufficient.

When the TodoImage build configuration is triggered, TeamCity makes sure all the dependencies for this build configuration are up to date. Consequently, all the dependent build configurations that form a build chain via the snapshot dependencies will be added to a build queue. The result is visible on the Build Chains tab of the project:

06-tc-pipelines-todo-backend-build-chain

Triggering the build chain

There are various kinds of triggers that are possible to set up for a build configuration in TeamCity. The VCS trigger is the one that reacts to the changes in a version control system.

11-tc-pipelines-todo-backend-vcs-trigger

In our example, there are two build configurations but we will only need one VCS trigger. This is because it’s possible to tell the trigger to monitor for the changes in the dependencies. In the configuration dialog, we have to enable “Trigger a build on changes in snapshot dependencies” checkbox.

We can configure a dedicated VCS trigger for each and every build configuration in the chain. However, each trigger creates some overhead: the server has to allocate some cycles to maintain the trigger. Additionally, it’s easier to make changes to the settings if you only have one trigger to configure. Hence, we can just add a single VCS trigger to the very last configuration in the build chain and it will do the job.

Re-using the results

The build chain completed successfully. We can start the build again. However, this time TeamCity will be smarter and skip running the builds for the dependencies where it detects no source changes.

For instance, on the first run, the build number for both our build configurations is 1. Now let’s try running the TodoImage build configuration a couple of times. Since there are no changes to TodoApp’s sources its build is not started and TeamCity re-uses the result of the 1st build. The TodoImage build number is now 3 since it was our intent to execute it.

07-tc-pipelines-todo-backend-build-chain-2

It is also possible to enforce building the dependencies even if there are no source changes. For this, click on the ellipsis next to the Run button of the build configuration. From the dialog, choose the Dependencies tab, and select the dependencies that are required to run.

08-tc-pipelines-todo-backend-enforce-dependcies

Configuring checkout rules

The current setup of our build chain doesn’t really provide us with much value yet. All the sources are located in the same GitHub repository, hence if we configure the trigger to fire on code changes both build configurations will run.

For instance, if we make a change to the Dockerfile we don’t need to run the build for the application. However, with the current setup, TeamCity detects that there’s a change in source control repository for TodoApp as well and both build configurations will execute incrementing the build numbers.

09-tc-pipelines-todo-backend-dockerfile-change

Instead, it would be nice to run the build only when necessary.

In this example, we would like to build TodoApp if there is a change to anything but the Dockerfile. And we only want to build a new image if just the Dockerfile changes – there’s no need to build the application in this case.

It is possible to configure the checkout rules for the VCS root accordingly. The checkout rules affect the actual checkout of the sources on the agent. That said, if we exclude any folders of the repository using the filters in checkout rules, those folders won’t end up in the workspace.

We will exclude the docker/ folder from the checkout in the TodoApp build configuration. And we will only pull the docker/ folder in the TodoImage build configuration, and ignore everything else.

10-tc-pipelines-todo-backend-chckout-rules-docker

Now, if we only change the Dockerfile, only TodoImage should execute and re-use the results from the previous TodoApp run without executing it.

Summary

The concept of snapshot dependencies is essential for configuring the build pipelines, aka build chains in TeamCity. The feature allows implementing incremental builds by re-using the results of the previous executions, thus saving a lot of time. In this article, we have also learned about triggering and VCS root checkout rules – both very useful for working with build chains.

Next, we will look at what else is possible with the features we have described – running the builds in parallel and also how to build projects from multiple source repositories.

Posted in Features, How-To's | Tagged , | Leave a comment

TeamCity 2019.2 EAP2 is out

We continue working on TeamCity 2019.2, and you can be the first to try some new features in this second EAP.

Since this EAP:

  • TeamCity provides server metrics in a popular Prometheus format, ready for monitoring on visual dashboards like Grafana.
  • Updates in experimental UI: access the Favorite Projects page from the sidebar, see the reason behind the queuing behavior for each build, and use the revamped Changes pop-up menu.
  • Our REST API provides the same information about cloud integration as that available in the TeamCity UI.

Kanpur 2019.2 EAP2 resolves over 100 issues. See the full list of fixes and features in our release notes.

Remember that the new release changes the TeamCity data format and downgrading to the previous production version is not supported. We recommend installing this EAP version on a trial server. If you installed the previous TeamCity EAP version, you can easily upgrade with our automatic update. In other cases, you can download the EAP build or pull the Docker image with the eap tag.

As with any EAP version, this one comes with a 60-day Enterprise evaluation license for an unlimited number of agents and build configurations.

We encourage you to try the new features and share your feedback with us, on our forum or issue tracker.

Happy building!

Posted in EAP, Features, Uncategorized | Leave a comment

Important Security Notice: XSS vulnerability allowing RCE

What happened

On July 24, 2019, a security researcher reported a security vulnerability in JetBrains TeamCity. The vulnerability allows cross-site scripting (XSS) on many pages, potentially making it possible to send an arbitrary HTTP request to the TeamCity server under the name of the currently logged-in user. The vulnerability affected TeamCity versions 2019.1 and 2019.1.1. It was fixed in TeamCity 2019.1.2.

What information was compromised

This security issue affected all TeamCity installations of versions 2019.1 and 2019.1.1, possibly allowing an unauthorized person to execute code remotely.

We do not have any information to confirm whether your particular TeamCity installation was compromised or not.

What actions we’ve taken

We resolved the issue as a regular XSS on July 18, 2019, and released the build of TeamCity with the fix on July 31, 2019. We are also adding automated tests to check for this type of vulnerabilities whenever changes are deployed to the codebase.

What actions you should take

Please upgrade to the latest build from our website.

Note: if you’re using a TeamCity version older than 2019.1, you’re not affected by this particular XSS. However, from a security point of view, upgrading to 2019.1.2 is still recommended because TeamCity 2019.1 contains other security fixes.

If you have any questions, please contact our Support. Please note that because of the nature of the vulnerability, we will not be able to disclose any details of how it can be exploited as this can affect other TeamCity installations.

Posted in Uncategorized | Tagged | Leave a comment

TeamCity 2019.1.3 is released

We have just released TeamCity 2019.1.3: this bugfix update resolves 50 problems.

Some users were facing issues with how the Pull Requests build feature handles old and reopened pull request branches. Now, builds will run properly on such branches and new commits will be consistently detected in them.

To prevent Linux file permissions conflicts for the Docker Compose step, if a build with this step is running on a Linux agent, it will now try to restore the ownership of files under own directories for the user who launched this agent (even if the build was interrupted).

The full list of fixes is available in our release notes.

We recommend upgrading to the new version, as this build addresses multiple security and performance issues.

As a regular bugfix update, TeamCity 2019.1.3 has the same data format as all 2019.1.x versions, which allows easily upgrading/downgrading within these builds.

Remember to check our Upgrade Notes and use one of these options to upgrade:

Your feedback is welcome in our forum or tracker.

Happy building!

Posted in Bugfix, Release, Uncategorized | Leave a comment

Test Automation with TeamCity

TeamCity offers a number of ways to help you work with tests. First of all, it presents the test results for each build execution so it’s easy to analyze what happened and why something has failed. Besides that, TeamCity includes advanced features like code coverage, flaky test detection, metadata in tests, and the ability to re-order the test execution. In this blog post, we provide a brief overview of these features.

Test results

It all starts with presenting the results. On the build configuration home page, for each build, we can see the number of tests that were executed. We can also see the numbers for failed, ignored, and muted tests.

01-tc-test-results

If we choose a single build and navigate to the build overview page, we will see some more data. For instance, for build number 14 we can see that there are 39 passed tests, and 1 test (system.CrashControllerTests) is ignored.

02-tc-test-overview

Further on, the Tests tab displays the full list of the tests that were executed:

03-tc-test-tab

It is possible to use the controls at the top of the list for grouping and filtering the results. Since TeamCity preserves the history of the tests, we can also see the historical data for their duration:

04-tc-test-tab-history

The duration is associated with the build agent on which the test was executed. The graph immediately shows us whether any agents are performing poorly for the specific test. We may want to fix something in the environment for that agent, or we can also exclude the agent entirely for this build configuration by adjusting the Agent Requirements.

Flaky test detection

Sometimes, seemingly at random, the same test of the same code will produce different results. We call those tests flaky. Flaky tests can be caused by various factors: environment affecting the test or application code, concurrency issues, etc.

It’s really great if we can detect such unstable tests sooner and fix them faster. Analyzing the historical test reports manually to find out, if there are any flaky tests would take ages. TeamCity detects flaky tests and displays them on the dedicated tab for a given project.

13-tc-test-flaky-tab

For every single test, we can view the history, assign an investigation, or temporarily mute the failure. For instance, we can mute all the flaky tests for the duration while someone is fixing them. Muting the tests will allow the build to pass so that the build chain can continue the execution. Once the tests are fixed, we can unmute them again.

07-tc-test-investigate-mute

Here’s an example of a flaky test. At the screenshot below, we can see that the test status was changing from passed to failed and back a few times, even though there were no related changes in the version control. Hence, the test was marked as flaky so that we could easily find it in the test report.

14-tc-test-flaky-history

The test results are associated with the build agent which helps to diagnose the failures better. For instance, the test might be failing only on a specific build agent. Then maybe the failure is related to the environment and not the test logic itself.

Code coverage

Code coverage is a combination of metrics that measure how your code is covered by unit tests. To get the code coverage information displayed in TeamCity for the supported tools, first we need to configure it in the dedicated section of a Build Runner settings page.

For instance, for Maven build step, at the bottom of the configuration page, we can find the Code Coverage section. We can choose the code coverage runner and specify which binaries should be inspected.

09-tc-test-coverage-maven-settings

After the test execution, the coverage metrics are presented on the build overview page. We can view the full report on the Code Coverage tab.

10-tc-test-coverage-overview

11-tc-test-coverage-packages

12-tc-test-coverage-source

Metadata in tests

Sometimes you need to provide more data with the test results. For instance, include a screenshot, add some numeric data, or just provide any description. This is useful when we need to analyze why the test failed and any additional information can help save precious time.

In TeamCity, a test may be associated with some information – the metadata. The types of data that we can associate with the test is text, numeric values, images, and files that are published as build artifacts.

We can use service messages to associate the data with the test. The data is then rendered with the test results:

15-tc-test-metadata-2

For more about metadata in tests, read this blog post.

Risk group tests re-ordering

A test suite with a large number of tests takes some time to execute. Sometimes, for large suites, the execution time can be many hours. What if the very last test fails and we want to rerun the test suite? Would we need to wait a few more hours just to learn that the same test fails again? Instead, it would be nice to run the recently failed tests first.

TeamCity provides a feature for running the risk group tests first. The feature works with TestNG, JUnit, and NUnit test suites and is supported for Ant and IntelliJ IDEA runners.

Posted in Blogroll, Features | Tagged , , , | 6 Comments