Fail build on metric change in TeamCity 7.0

One of TeamCity’s remarkable features is a possibility to analyze the code of your application, find duplicates, show corresponding reports and charts. You can calculate code coverage metrics, run code inspections, and see results right in the TeamCity’s UI. And in most cases, these features work out of the box, without setting up additional external plugins.

All these code examining tools generate various numeric metrics, and you can see the dynamic of their change in the TeamCity UI.

What you could not do before (well, almost could not) is to specify metric thresholds, which would fail the build upon exceeding them. For instance, you couldn’t tell TeamCity that the build must fail if code coverage or code duplicates number is worse than in the previous build.

Corresponding requests stayed for a while in our tracker and collected quite a few number of votes:
- TW-2539 Inspection Build: fail if more inspection errors than in last successful build
- TW-3040 Make build fail if code coverage is below certain threshold
- TW-3041 Make build fail if code coverage is below previous code coverage value
- TW-4953 Setting to fail a build when number of found code duplicates exceeds some threshold
- TW-15679 Option to fail a build if a specific statistics value is below/greater then in last successful/tagged build

When we started pondering over the task, it became obvious, that it can be generalized. And besides code quality, we could consider other metrics, such as number of tests in a build or volume of published artifacts:
- TW-12449 Option to fail a build if test number drops
- TW-5451 Option to fail a build when not all artifacts are published (artifacts validation)

Moreover, if a build publishes its own numeric metric (and it is very easy to set up), this metric can also be used to determine build status.

So now you can specify several metric-based build failure rules for a build configuration:

or

As you see, you can create rules for both the absolute metric values, and for the difference with a previous build in the same build configuration. As a previous build anchor, you can use:
- last finished build
- last successful build
- last pinned build
- build with a specified build number
- last finished build with given tag

The default list of available metrics is:
- artifact bytes (you can fail a build, if not all artifacts were published, basing on their volume)
- number of tests in a build
- number of ignored tests
- code inspections: errors, warnings
- code duplicates count
- code coverage percentage – lines, blocks, methods, classes

To add your custom metric to this list, you have to change TeamCity’s configuration file main-config.xml (<TeamCity data directory>/config/main-config.xml) and add the following section there:

<build-metrics>
<statisticValue key="my_file_count" description="number of files"/>
</build-metrics>

So, if your build will publish value my_file_count, you can use it as a criteria for a build failure.

When build failure metrics are set up, the interface will look like:

When a build fails because of the bad metric value, it looks like this:

The feature is not completed yet, so your comments and feedback are very much welcome. To try it live, install TeamCity 7.0 EAP build, which is about to be released in the nearest future.

This entry was posted in EAP, Features, Tips&Tricks. Bookmark the permalink.

4 Responses to Fail build on metric change in TeamCity 7.0

  1. Peter Mounce says:

    Awesome!

    We work on a legacy code-base, so having the ability to effectively halt entropy like this (fail build if we introduced something worse) will be great.

    Will we be able to bless a build (as in, not just pass it, but highlight it as great work)? This kind of thing could play into a nice bit of rivalry, because you could introduce some kind of medals system like stackoverflow.com for achievements while at work:
    * person has most builds this week/month that were green
    * person added the most tests this week/month
    * person added the most _reliable_ tests this week/month
    * person added the best tests this month based on the fact that other people caused them to fail (presumably) highlighting defects before production
    * team (group) eliminated the most code inspection failures this week/month
    * team (group) added the most coverage due to unit/integration/functional/UI tests added this month
    * team (group) sped up the build the most this month

    (person/group and time-period would be configurable, and there’d be a set of pages to show by-day/week/month/all-time, as well as add medals/badges to individual profile pages and things like that)

    We’re a .NET shop, so will we be able to integrate with tools like NDepend and Nitriq to fail based on code-query rules? Or, even, various ReSharper patterns from that command-line inspection-runner that must surely be in the works?

  2. KIR says:

    Hello Peter,

    Thanks for the feedback!

    The idea with rewards/medal system is really funny and interesting – never thought about this in the context of TeamCity.

    Regarding integration with NDepend or Nitriq – if you create build configurations which would publish any metric values, you can use these metric values to create failure conditions.

    And yes, ReSharper inspection patterns are in the works :)

    Thanks again,
    KIR

  3. xumix says:

    Is there any way to add additional parameter to VCS commandline executables? e.g. I want to add –uncompressed to my Mercurial Root, so it will be much faster to transfer changes/clones to my agents.

  4. Dmitry says:

    Hi Xumix,
    we have a checkbox “Use uncompressed transfer” in the settings of hg VCS root.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>