TeamHackCity: Plugin Week 2014

Hello, one and all!

Today I want you to take a peek behind the scenes of the TeamCity development process. Some time ago our team of superstars held the TeamCity plugin week which has by now become traditional. The period in-between releases seemed to be the perfect time for our developers to unleash their creativity and boy, they had a ball!

We hope to arouse your interest, but be warned: some of the plugins are pretty raw, in pre-alpha state, and PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND. Do not install them in production without testing!

Being given time to put their thoughts into lines of code, our developers went really wild, and came up with a number of ideas! Prepare yourselves for the unexpected!

SonarQube Plugin

We have a lot of votes for Sonar integration in our tracker. So please welcome the initial version of the Sonar plugin! The plugin comes with the Sonar build step and a simple user interface for configuring Sonar servers. More will be added based on your feedback!

Sonar Build Step
Author: Andrey Titov
Availability: Sources on GitHub

The TeamCity Achievements plugin

A sense of humor is a sign of intelligence, and this plugin is indispensable for TeamCity as an intelligent Continuous Integration server! It’s all about challenging and motivating your developers: the plugin grants achievements to users for record numbers of various actions in the TeamCity web interface.

achievements

  • You will be bestowed the proud name of a Saboteur for bombing (i.e. muting) several tests with automatic explosion (unmute) on a specific date
  • Those disarming (unmuting) several tests, will be called brave Sappers
  • You’ll see how many Novelists you have with this achievement granted to the code contributors for extra-long commit descriptions.

Author: Pavel Sher
Availability: Sources on GitHub
Continue reading

Posted in Blogroll, Features, FYI, News & Events, Uncategorized | Leave a comment

TeamCity: Beyond Continuous Integration, Webinar Recording

The recording of our June 26th webinar with Wes McClure, TeamCity: Beyond Continuous Integration, is now available on JetBrains YouTube Channel.

In this webinar, we explore the landscape of opportunities beyond Continuous Integration. See how build chains and dependencies provide a robust framework to orchestrate the benefits automation can bring, to the software you create.

The prior webinar mentioned in the recording is Getting Started with TeamCity. Wes’s slides are available on SlideShare.

Below are select questions and answers from our webinar.

Continue reading

Posted in How-To's, Partner, Tips&Tricks | Tagged , , , | 1 Comment

TeamCity: Beyond Continuous Integration, June 26th Webinar

Join us Thursday, June 26th, 15:00 – 16:00 GMT (11:00 AM – 12:00 PM EDT) for our free webinar, TeamCity: Beyond Continuous Integration with Wes McClure.

Continuous Integration is a gateway to further automation. It lays a foundation to change software quickly, with confidence. But why stop there? As we experiment with automation, we can establish techniques to further test, distribute and deliver software. Can you imagine what it would be like to focus on adding valuable features without being burdened with how they’re released to the world?

Come experiment with us as we explore the landscape of opportunities beyond Continuous Integration. See how build chains and dependencies provide a robust framework to orchestrate the benefits automation can bring, to the software you create.

Space is limited, please register now. There will be an opportunity to ask questions during the webinar.

About the Presenter:

Wes McClureWes McClure is passionate about helping companies achieve remarkable results with technology and software. He’s had extensive experience developing software and working with teams to improve how software is developed to meet business objectives. Wes launched Full City Tech to leverage his expertise to help companies rapidly deliver high quality software to delight customers. He has a strong background in using Continuous Integration with TeamCity to bring quality to the table.
Posted in How-To's, Partner, Tips&Tricks | Tagged , , , , , | Leave a comment

A Step by Step Guide: Create a custom TeamCity MetaRunner to notify NewRelic that a new deployment has occurred.

This guest blog post is brought to you by Howard van Rooijen of endjin, a JetBrains Training and Consulting Partner. Your feedback is welcome in the comments section. 

As systems grow in size and complexity, without rigorous Application Lifecycle Management processes in place, organisations risk degrading into a state of chaos and fear. Earlier in the year I wrote a whitepaper for JetBrains, “From Chaos, through Fear, to Confidence” which describes how endjin helped one of our customers regain their confidence in their development process through the introduction of engineering practices, automated processes and feedback loops that have become the main tenets of the DevOps movement:

devops-feedback-loops

It is very rare to find a “one size fits all” solution for this type of process automation, but a more pragmatic approach of picking the “best of breed” tooling in each area and stitching them together into a cohesive solution can yield great results. Integration and extensibility points become key; luckily TeamCity and YouTrack have plenty of both.

In the previous diagram there is one important feedback loop missing:

teamcity-notify-newrelic

We’re not notifying NewRelic that a new deployment has occurred. One of the main concerns of operational monitoring is checking to see that a change to any production system doesn’t negatively impact performance or reliability, and the most likely time this will happen is when a deployment has occurred.

For those of you who are unfamiliar with NewRelic, it’s a cloud based application monitoring platform that allows you to keep tabs on the CPU & memory utilisation, number of web transactions, unhandled errors and user-definable transactions within your application. We have used NewRelic on a number of our Microsoft Azure projects, from the development environment, through test and UAT and production. It’s an invaluable tool gaining application telemetry if you deploy into a cloud environment, where you don’t really have access to the operating system.

NewRelic has a concept of a deployment as an application event, but requires this to be raised by a 3rd party. By default the charts are unadorned by deployment information and the deployment event stream is empty:

new-relic-no-deployments

This guide describes how to take a pre-existing tool, wrap it in a tool-based MetaRunner with a custom user interface, which can be used by TeamCity to notify NewRelic that a deployment has occurred:

new-relic-deployment-events

Once NewRelic is notified a deployment has happened, new diagnostic features are enabled to help analyse deployment related performance changes.

The NewRelic Deployment API

Fortunately NewRelic have enabled deployment notification integration via a simple ReST API which you can push data to via the following command with Curl:

curl -H "x-api-key:REPLACE_WITH_YOUR_API_KEY" -d "deployment[app_name]=REPLACE_WITH_YOUR_APP_NAME" https://api.newrelic.com/deployments.xml

The API has the following options, which you should apply via the –d option of Curl:

Exactly one of the following is required:

  • deployment[app_name]: The value of app_name in the newrelic.yml file used by the application OR
  • deployment[application_id]: The application id, found in the URL when viewing the application in RPM.

Following are optional parameters:

  • deployment[changelog] or deployment[changes]: A list of changes for this deployment
  • deployment[description]: Text annotation for the deployment– notes for you
  • deployment[revision]: A revision number (e.g., git commit SHA)
  • deployment[user]: The name of the user/process that triggered this deployment
  • deployment[appname]: Name of the application
  • deployment[environment]: The environment for this deployment

This integration could be delivered via  a customized build script, but a more user friendly and reusable approach is using one of TeamCity’s best kept secrets: tool-based MetaRunners.

TeamCity MetaRunners

Maarten Balliauw has written a great introductory post on creating MetaRunners and I would strongly recommend you read that post to give you a good grounding in what they are and what they are capable of. JetBrains have also created a great resource for people interested in developing MetaRunners – the MetaRunner PowerPack on GitHub, which demonstrates a many different approaches across the different technology stacks.

In essence, a MetaRunner is a user-definable set of build steps, which can either be written from scratch by hand or extracted from existing, configured build steps, which you can parameterize and then create a simple user interface over, using primitive controls (textbox, checkbox, dropdown list) and validation, to enable easy re-use. In this case a MetaRunner is simply an xml file, which contains the definition of the build steps and also an embedded version of scripts you wish to execute.

MetaRunners can be used in another (as yet, undocumented) way; by following a series of conventions, you can package up tools and scripts, create a custom UI definition to allow easy reuse, package them up as a zip and deploy on to your TeamCity server. This type of tool-based MetaRunner plugin will become available as a build runner via the TeamCity UI.

1. Creating the tool-based MetaRunner Plugin

The structure for a MetaRunner plugin is as follows:

Plugin
|   teamcity-plugin.xml
|
+—agent
|   |   teamcity-plugin.xml
|   |
|   \—bin
|           Curl-License.txt
|           curl.exe
|
\—server
\—metaRunners
MRPLUGIN_NewRelicDeploymentNotifier.xml

At the root level of the plugin there is a teamcity-plugin.xml file. This contains all the information that will be displayed in the TeamCity Build Runner UI (i.e. name, description etc.)

The second teamcity-plugin.xml located file under the agent directory is simpler – it is just a simple descriptor which signifies that the plugin is a tool package (i.e. it contains binaries and scripts you wish to include with the MetaRunner):

All of the tooling files you wish to include should also be stored under the agent directory (I have created a bin sub-folder, in this case it’s not really needed, but if you want to package up lots of tools / files, it’s simpler to keep the tools separate from the mandatory plugin files).

The final required file called MRPLUGIN_NewRelicDeploymentNotifier.xml and is located in server/metaRunners:

This file is quite complex at first glance, but is actually quite trivial. The document contains two main sections; parameters, which defined the parameters you allow users to configure via the UI, that will be passed into the context of the MetaRunner upon execution and build-runners which described the build runner you want to execute (the two most straight forward are; ‘simpleRunner’ which equates to running within the shell environment and ‘jetbrains_powershell’ which allows you to run a PowerShellScript). Note how we just reuse the text from the NewRelic Deployment Notification documentation for the description fields for the UI. In this example we are using a simpleRunner to wrap the required Curl command from the NewRelic ReST API inside the script.content element:

As you can see, the parameters we defined in the parameters section can be referenced inside the script.content element by using %your.custom.parameter%. One gotcha is to ensure that you need to wrap the path to the tool and its parameters with " characters to ensure paths and values are resolved correctly. One important item to note is the path to the tool .exe:

"%teamcity.agent.tools.dir%\nr-deployment-notifier\bin\curl.exe"

%teamcity.agent.tools.dir% is a standard TeamCity environment variable, but nr-deployment-notifier is derived from the name of the tool package zip file you create from the agent directory (see steps below).

2. Packaging the tool-based MetaRunner Plugin

Now that you have all the configuration files and the tools you want to include, you need to package them up in a zip file. Firstly, you need to package up the contents of the agents folder, as this zip file will be distributed to all build agents by TeamCity. I originally wrote a build script which automated this process, but there seems to be some incompatibility between 7Zip and the Java APIs that TeamCity uses to unpackage the zip file, so you need to resort to using the built in zip features of Windows:

package-metarunner-01-package-tools

As mentioned above, the name you give this zip file is important as the contents will be unpackaged into a folder of the same name, which you need to refer to in your script.content element:

package-metarunner-02-name-tools-package

You can delete the rest of the files, so just the zip is located in the agents directory (obviously, do this to a copy of your files, not the originals):

package-metarunner-03-delete-original-files

Next you need to package the whole plugin directory the same way:

package-metarunner-04-package-plugin

The name of this file doesn’t have any repercussions, so any name will do:

package-metarunner-05-name-plugin-package

3. Installing the MetaRunner Plugin

Next you need to install the zip into the main TeamCity plugins directory, located at <TeamCity Data Directory>/plugins. The screenshot below shows a single server TeamCity environment (where both the Build Agent and TeamCity Server  are located on the same OS). The left hand windows shows the Build Agent Tools folder located at <TeamCity Home>/buildAgent/tools (where the MetaRunner package will be automatically unpacked) and on the right hand side the <TeamCity Data Directory>/plugins folder.

teamcity-install-metarunner-plugin

Once you have dropped the plugin into the <TeamCity Data Directory>/plugins folder, after a few minutes a new folder, with the same name as the zip you created inside the agents directory, should appear. If the new directory doesn’t appear after two minutes, you may need to restart the Build Agent service.

teamcity-metarunner-plugin-installed

4. Using the MetaRunner Plugin

Now that the plugin has been successfully deployed to the TeamCity server, you should be able to create a new build step and select the “NewRelic Deployment Notifier” as the Runner type. You should now be able to see the rest of the UI you have defined:

teamcity-build-step-configuration

The two mandatory fields are the API Key and the App Name of the app that has been deployed. The remaining fields are optional, but you can add the standard TeamCity environment variables %build.number% to automatically pass in the build number of the deployment that’s occurred and %teamcity.build.triggeredBy% to pass in the name of the user who kicked off the deployment.

If the build step runs successfully, you should see the following output:

teamcity-metarunner-succeeded

Side Note: in order to interact with HTTPS services, Curl has a dependency on OpenSSL. In order to use this MetaRunner, you will need to install OpenSSL on each of your Build Agents. If you do not have this installed, the above build step with fail with an error message like: Process exited with code –1073741515. More information about how I diagnosed this requirement in the “Troubleshooting the MetaRunner Plugin” section below.

The end result

With the deployment notifier properly configured, every time a deployment is carried out by TeamCity, NewRelic with visualize the deployment as a vertical line across its graphs. If you hover over the line with your mouse, the details you passed into the notifier are displayed:

new-relic-deployment-successful

Once you have notified NewRelic of a deployment, some new features appear, such as the ability to analyze performance since the last deployment:

new-relic-performance-since-last-deployment

Troubleshooting the MetaRunner Plugin

Once I deployed the original SimpleRunner version of the MetaRunner plugin, I saw the following “Process exited with code –1073741515″ error. To try and diagnose the issue, I wanted to peek into the temporary .cmd file (highlighted in yellow) that TeamCity had generated to ensure that my script was properly written, but unfortunately, the file is deleted as soon as it has been run.

teamcity-metarunner-failed

So I created a new PowerShellRunner version of the MetaRunner plugin, that allowed me to add more validation and debugging logic:

teamcity-debugging-powershell-metarunner

This all seemed correct, but unlike the SimpleRunner, when I navigated to the location highlighted in yellow, the file still existed, so I opened it in PowerShell ISE and ran it and saw a Windows exception dialog with the following message from Curl:

The program can't start because libsasl.dll is missing from your computer.

Try reinstalling the program to fix this problem.

Installing OpenSSL solved the problem, but I thought the fact that the .PS1 script does not appear to get deleted by TeamCity 8.x could be a useful technique for anyone else trying to diagnose why their MetaRunner is failing.

Hopefully this guide has demonstrated that it is straightforward to develop a custom tool-based MetaRunner and it will encourage you to think about other reusable tools you could expose to TeamCity in the same way. For example, we wrap our Microsoft Azure Deployment Tooling in a MetaRunner to allow any endjineer (or customer) to deploy into Azure following our conventions by filling in a few simple fields relevant to their project in the TeamCity UI.

The source for both the SimpleRunner and PowerShellRunner versions of the plugin are available on GitHub.

Work Smarter, Not Harder.

@HowardvRooijen | @endjin

endjin logoEndjin is a strategy, experience, development & cloud consultancy. Established in 2010, endjin has become the number one Microsoft Azure Partner in EMEA, delivering projects using cloud, mobile, desktop and embedded technologies in retail, financial services, and consumer applications. The company mantra is to “work smarter” and we help our customers do the same by improving their processes & engineering practices.
Posted in Features, How-To's, Partner, Tips&Tricks | Tagged , , , , , | 3 Comments

Webinar Recording and Q&A: .NET Code Coverage for Continuous Integration using TeamCity and dotCover

The recording of our June 10th webinar with Maarten Balliauw, .NET Code Coverage for Continuous Integration using TeamCity and dotCover, is now available on JetBrains YouTube Channel and slides on slideshare.

In this webinar we use dotCover to collect code coverage information while running tests in our CI process. We can see how we can configure code coverage and how we can use the TeamCity Visual Studio plugin to download the coverage snapshot generated on the build server and inspect it using dotCover on a developer machine.

How much of our code is being covered by our unit tests? Are there areas we are not testing? By capturing code coverage data during a test run, we can analyze which areas of our applications are well-tested and which ones require additional tests to be written. And where better to capture code coverage information than on our build server? Find out in this webinar.

Below are select questions and answers from our webinar:

Q: Can we use MStest with TeamCity and also get code coverage with dotCover?

A: Yes, test runner type and .NET coverage tool can be chosen in TeamCity Build Configuration Settings tab. Please check the following settings:
MSTest and dotCover with TeamCity

Q: Is it possible to use dotCover on TeamCity with xUnit.net tests?

A: Yes, it is possible. If you are using xUnit and you want to capture code coverage, you can run dotCover from the command line and import the data into TeamCity by using a service message. The process details are described by Maarten during the webinar (starting from 24th minute).

For more information on dotCover, please visit dotCover official website.

Maarten BalliauwMaarten Balliauw is a Technical Evangelist at JetBrains. His interests are all web: ASP.NET MVC, PHP and Windows Azure. He’s a Microsoft Most Valuable Professional (MVP) for Windows Azure and an ASPInsider. He has published many articles in both PHP and .NET literature such as MSDN magazine and PHP architect. Maarten is a frequent speaker at various national and international events such as MIX (Las Vegas), TechDays, DPC and others.
Posted in How-To's, Tips&Tricks | Tagged , , , , , | Leave a comment

TeamCity 8.1.3 (build 30101) is here!

Dear TeamCity users, greetings to all of you! We are happy to announce the immediate availability of TeamCity 8.1.3! Build 30101 is mostly a bugfix release containing about 100 fixes.

Our users running TeamCity on an Oracle database will be relieved to learn that we have fixed the unpleasant issue with the high memory usage by the server.

We have good news for everyone using TeamCity and Git via TFS! The Git plugin for TeamCity now supports NTLM over HTTP authentication, so we easily integrate with Git hosted from Team Foundation Server 2013. Note that the Active Directory credentials need to be specified in the authentication settings of your Git VCS root.

This release also comes with several important performance fixes. If collecting changes in build chains is slow, or your server CPU usage is high, we strongly recommend upgrading to 8.1.3: chances are these problems have been fixed by now. A number of other important fixes is included as well, see our Release Notes for the complete list.

Finally, while working on the problem with a corrupted NuGet feed, we implemented NuGet packages re-indexing in this release. Now, on the server startup, TeamCity determines which packages are missing in the feed, and starts re-indexing these packages. This process may take some time, and as a result, after upgrade to this release, the number of packages in your NuGet feed can increase.

As with any bugfix release, 8.1.3 does not change the data format, so in case of any problems, you can easily revert to a previous 8.1.x version.

If you still have not switched over to TeamCity 8.1, check out all the new features to see what you are missing out on! Make sure you read our Upgrade Notes and do upgrade to the latest version of TeamCity!

Download TeamCity 8.1.3 now and happy building!

Posted in Bugfix, EAP, Features, FYI, News & Events | Leave a comment

Maximize Outcomes by Establishing a Release Driven Workflow, Upfront

This guest blog post is by Wes McClure of Full City Tech Co., a JetBrains Training and Consulting Partner. Your thoughts & feedback are welcome in the comments section. 

What if you never had to worry about another software release? What if you had confidence in frequently delivering valuable software? What would it be like to establish this before you ever write the first line of code? Find out how focusing on outcomes and working backward to establish a release driven workflow, upfront, can make this a reality:

Download Now: Maximize Outcomes by Establishing a Release Driven Workflow, Upfront [PDF, no registration required]

If you’re stuck trying to reap the benefits of automation, and have yet to experience the transformative impact on how you develop and release software, this article has the strategy for you to get started. After years of working with automation, I’ve noticed many teams struggle to effectively adopt the practice of Continuous Integration, let alone the myriad of awesome beyond. Although Continuous Integration is a great starting point that leads to further automation, sometimes it’s hard to visualize the potential value yet to be claimed.

Instead of taking it one step at a time, the time has come to prioritize what it takes to release. The multitude of practices, frameworks and tools to automate everything under the sun are well documented and readily available. We should use them to build the foundation of a process we can evolve to provide confidence, rapid feedback and support. With this new found certainty, we’ll only be limited by our imagination.

Dmitri NesterukWes McClure is passionate about helping companies achieve remarkable results with technology and software. He’s had extensive experience developing software and working with teams to improve how software is developed to meet business objectives. Wes launched Full City Tech to leverage his expertise to help companies rapidly deliver high quality software to delight customers. He has a strong background in using Continuous Integration with TeamCity to bring quality to the table.
Posted in How-To's, Partner, Tips&Tricks | Tagged , , | Leave a comment

Live Webinar: .NET Code Coverage for Continuous Integration using TeamCity and dotCover, June 10th

UPDATED: Join us Tuesday, June 10th, 14:00 – 15:00 GMT (10:00 AM – 11:00 PM EDT) for our free live webinar, .NET Code Coverage for Continuous Integration using TeamCity and dotCover, with Maarten Balliauw.

How much of our code is being covered by our unit tests? Are there areas we are not testing? By capturing code coverage data during a test run, we can analyze which areas of our applications are well-tested and which ones require additional tests to be written. And where better to capture code coverage information than on our build server?

In this webinar we will use dotCover to collect code coverage information while running tests in our CI process. We will see how we can configure code coverage and how we can use the TeamCity Visual Studio plugin to download the coverage snapshot generated on the build server and inspect it using dotCover on a developer machine.

Space is limited, please register now.

About the Presenter:

Maarten BalliauwMaarten Balliauw is a Technical Evangelist at JetBrains. His interests are all web: ASP.NET MVC, PHP and Windows Azure. He’s a Microsoft Most Valuable Professional (MVP) for Windows Azure and an ASPInsider. He has published many articles in both PHP and .NET literature such as MSDN magazine and PHP architect. Maarten is a frequent speaker at various national and international events such as MIX (Las Vegas), TechDays, DPC and others..
Posted in How-To's, Tips&Tricks | Tagged , , , | 1 Comment

Webinar Recording: Getting Started with TeamCity

The recording of our April 7th webinar, Getting Started with TeamCity, is now available on JetBrains YouTube Channel.

Learn how to get started with TeamCity for implementing continuous integration with Wes McClure of Full City Technology Co. In addition to the video there’s a wealth of information in the Q&As section below.

Continue reading

Posted in Features, How-To's, Partner, Tips&Tricks | Tagged , , , , | 4 Comments

TeamCity 8.1.2 is Out

During the past month we’ve been focusing on making TeamCity more stable and bug-free, and today we are pleased to announce that TeamCity 8.1.2 has just been released!

As hinted by the version number, this is a bugfix and tweak build resolving about 80 issues, many of which we were able to investigate and fix thanks to your feedback!

The issue with notification popups in the Windows tray notifier introduced in 8.1 has been resolved, besides, this release comes with quite a few performance and usability enhancements. One of such improvements is the possibility to track memory and CPU usage by TeamCity with the new charts on the Diagnostics page of the Server Administration Web UI.

For .NET developers using JetBrains tools, this is an ideal moment to switch over to the new TeamCity build, as our VS Addin is now compatible with ReSharper 8.2. The Java code coverage engines used by TeamCity, IntelliJ IDEA and JaCoCo, now come with support for Java 8.

To take advantage of the fixes and to see why we consider the TeamCity 8.1.2 release to be the best version available, we encourage users of 8.1.1 and all prior versions to upgrade. Check out our Release Notes and download the new version now!

Happy Building!

Posted in Bugfix, Features, FYI, News & Events, Tips&Tricks | 2 Comments