A Step by Step Guide: Create a custom TeamCity MetaRunner to notify NewRelic that a new deployment has occurred.

Robert Demmer

This guest blog post is brought to you by Howard van Rooijen of endjin, a JetBrains Training and Consulting Partner. Your feedback is welcome in the comments section. 

As systems grow in size and complexity, without rigorous Application Lifecycle Management processes in place, organisations risk degrading into a state of chaos and fear. Earlier in the year I wrote a whitepaper for JetBrains, “From Chaos, through Fear, to Confidence” which describes how endjin helped one of our customers regain their confidence in their development process through the introduction of engineering practices, automated processes and feedback loops that have become the main tenets of the DevOps movement:

It is very rare to find a “one size fits all” solution for this type of process automation, but a more pragmatic approach of picking the “best of breed” tooling in each area and stitching them together into a cohesive solution can yield great results. Integration and extensibility points become key; luckily TeamCity and YouTrack have plenty of both.

In the previous diagram there is one important feedback loop missing:

We’re not notifying NewRelic that a new deployment has occurred. One of the main concerns of operational monitoring is checking to see that a change to any production system doesn’t negatively impact performance or reliability, and the most likely time this will happen is when a deployment has occurred.

For those of you who are unfamiliar with NewRelic, it’s a cloud based application monitoring platform that allows you to keep tabs on the CPU & memory utilisation, number of web transactions, unhandled errors and user-definable transactions within your application. We have used NewRelic on a number of our Microsoft Azure projects, from the development environment, through test and UAT and production. It’s an invaluable tool gaining application telemetry if you deploy into a cloud environment, where you don’t really have access to the operating system.

NewRelic has a concept of a deployment as an application event, but requires this to be raised by a 3rd party. By default the charts are unadorned by deployment information and the deployment event stream is empty:

This guide describes how to take a pre-existing tool, wrap it in a tool-based MetaRunner with a custom user interface, which can be used by TeamCity to notify NewRelic that a deployment has occurred:

Once NewRelic is notified a deployment has happened, new diagnostic features are enabled to help analyse deployment related performance changes.

The NewRelic Deployment API

Fortunately NewRelic have enabled deployment notification integration via a simple ReST API which you can push data to via the following command with Curl:

curl -H "x-api-key:REPLACE_WITH_YOUR_API_KEY" -d "deployment[app_name]=REPLACE_WITH_YOUR_APP_NAME" https://api.newrelic.com/deployments.xml

The API has the following options, which you should apply via the –d option of Curl:

Exactly one of the following is required:

• deployment[app_name]: The value of app_name in the newrelic.yml file used by the application OR
• deployment[application_id]: The application id, found in the URL when viewing the application in RPM.

Following are optional parameters:

• deployment[changelog] or deployment[changes]: A list of changes for this deployment
• deployment[description]: Text annotation for the deployment– notes for you
• deployment[revision]: A revision number (e.g., git commit SHA)
• deployment[user]: The name of the user/process that triggered this deployment
• deployment[appname]: Name of the application
• deployment[environment]: The environment for this deployment

This integration could be delivered via  a customized build script, but a more user friendly and reusable approach is using one of TeamCity’s best kept secrets: tool-based MetaRunners.

TeamCity MetaRunners

Maarten Balliauw has written a great introductory post on creating MetaRunners and I would strongly recommend you read that post to give you a good grounding in what they are and what they are capable of. JetBrains have also created a great resource for people interested in developing MetaRunners – the MetaRunner PowerPack on GitHub, which demonstrates a many different approaches across the different technology stacks.

In essence, a MetaRunner is a user-definable set of build steps, which can either be written from scratch by hand or extracted from existing, configured build steps, which you can parameterize and then create a simple user interface over, using primitive controls (textbox, checkbox, dropdown list) and validation, to enable easy re-use. In this case a MetaRunner is simply an xml file, which contains the definition of the build steps and also an embedded version of scripts you wish to execute.

MetaRunners can be used in another (as yet, undocumented) way; by following a series of conventions, you can package up tools and scripts, create a custom UI definition to allow easy reuse, package them up as a zip and deploy on to your TeamCity server. This type of tool-based MetaRunner plugin will become available as a build runner via the TeamCity UI.

1. Creating the tool-based MetaRunner Plugin

The structure for a MetaRunner plugin is as follows:

Plugin
|   teamcity-plugin.xml
|
+—agent
|   |   teamcity-plugin.xml
|   |
|   \—bin
|           Curl-License.txt
|           curl.exe
|
\—server
\—metaRunners
MRPLUGIN_NewRelicDeploymentNotifier.xml

At the root level of the plugin there is a teamcity-plugin.xml file. This contains all the information that will be displayed in the TeamCity Build Runner UI (i.e. name, description etc.)

The second teamcity-plugin.xml located file under the agent directory is simpler – it is just a simple descriptor which signifies that the plugin is a tool package (i.e. it contains binaries and scripts you wish to include with the MetaRunner):

All of the tooling files you wish to include should also be stored under the agent directory (I have created a bin sub-folder, in this case it’s not really needed, but if you want to package up lots of tools / files, it’s simpler to keep the tools separate from the mandatory plugin files).

The final required file called MRPLUGIN_NewRelicDeploymentNotifier.xml and is located in server/metaRunners:

This file is quite complex at first glance, but is actually quite trivial. The document contains two main sections; parameters, which defined the parameters you allow users to configure via the UI, that will be passed into the context of the MetaRunner upon execution and build-runners which described the build runner you want to execute (the two most straight forward are; ‘simpleRunner’ which equates to running within the shell environment and ‘jetbrains_powershell’ which allows you to run a PowerShellScript). Note how we just reuse the text from the NewRelic Deployment Notification documentation for the description fields for the UI. In this example we are using a simpleRunner to wrap the required Curl command from the NewRelic ReST API inside the script.content element:

As you can see, the parameters we defined in the parameters section can be referenced inside the script.content element by using %your.custom.parameter%. One gotcha is to ensure that you need to wrap the path to the tool and its parameters with " characters to ensure paths and values are resolved correctly. One important item to note is the path to the tool .exe:

"%teamcity.agent.tools.dir%\nr-deployment-notifier\bin\curl.exe"

%teamcity.agent.tools.dir% is a standard TeamCity environment variable, but nr-deployment-notifier is derived from the name of the tool package zip file you create from the agent directory (see steps below).

2. Packaging the tool-based MetaRunner Plugin

Now that you have all the configuration files and the tools you want to include, you need to package them up in a zip file. Firstly, you need to package up the contents of the agents folder, as this zip file will be distributed to all build agents by TeamCity. I originally wrote a build script which automated this process, but there seems to be some incompatibility between 7Zip and the Java APIs that TeamCity uses to unpackage the zip file, so you need to resort to using the built in zip features of Windows:

As mentioned above, the name you give this zip file is important as the contents will be unpackaged into a folder of the same name, which you need to refer to in your script.content element:

You can delete the rest of the files, so just the zip is located in the agents directory (obviously, do this to a copy of your files, not the originals):

Next you need to package the whole plugin directory the same way:

The name of this file doesn’t have any repercussions, so any name will do:

3. Installing the MetaRunner Plugin

Next you need to install the zip into the main TeamCity plugins directory, located at <TeamCity Data Directory>/plugins. The screenshot below shows a single server TeamCity environment (where both the Build Agent and TeamCity Server  are located on the same OS). The left hand windows shows the Build Agent Tools folder located at <TeamCity Home>/buildAgent/tools (where the MetaRunner package will be automatically unpacked) and on the right hand side the <TeamCity Data Directory>/plugins folder.

Once you have dropped the plugin into the <TeamCity Data Directory>/plugins folder, after a few minutes a new folder, with the same name as the zip you created inside the agents directory, should appear. If the new directory doesn’t appear after two minutes, you may need to restart the Build Agent service.

4. Using the MetaRunner Plugin

Now that the plugin has been successfully deployed to the TeamCity server, you should be able to create a new build step and select the “NewRelic Deployment Notifier” as the Runner type. You should now be able to see the rest of the UI you have defined:

The two mandatory fields are the API Key and the App Name of the app that has been deployed. The remaining fields are optional, but you can add the standard TeamCity environment variables %build.number% to automatically pass in the build number of the deployment that’s occurred and %teamcity.build.triggeredBy% to pass in the name of the user who kicked off the deployment.

If the build step runs successfully, you should see the following output:

Side Note: in order to interact with HTTPS services, Curl has a dependency on OpenSSL. In order to use this MetaRunner, you will need to install OpenSSL on each of your Build Agents. If you do not have this installed, the above build step with fail with an error message like: Process exited with code –1073741515. More information about how I diagnosed this requirement in the “Troubleshooting the MetaRunner Plugin” section below.

The end result

With the deployment notifier properly configured, every time a deployment is carried out by TeamCity, NewRelic with visualize the deployment as a vertical line across its graphs. If you hover over the line with your mouse, the details you passed into the notifier are displayed:

Once you have notified NewRelic of a deployment, some new features appear, such as the ability to analyze performance since the last deployment:

Troubleshooting the MetaRunner Plugin

Once I deployed the original SimpleRunner version of the MetaRunner plugin, I saw the following “Process exited with code –1073741515” error. To try and diagnose the issue, I wanted to peek into the temporary .cmd file (highlighted in yellow) that TeamCity had generated to ensure that my script was properly written, but unfortunately, the file is deleted as soon as it has been run.

So I created a new PowerShellRunner version of the MetaRunner plugin, that allowed me to add more validation and debugging logic:

This all seemed correct, but unlike the SimpleRunner, when I navigated to the location highlighted in yellow, the file still existed, so I opened it in PowerShell ISE and ran it and saw a Windows exception dialog with the following message from Curl:

The program can't start because libsasl.dll is missing from your computer.

Try reinstalling the program to fix this problem.

Installing OpenSSL solved the problem, but I thought the fact that the .PS1 script does not appear to get deleted by TeamCity 8.x could be a useful technique for anyone else trying to diagnose why their MetaRunner is failing.

Hopefully this guide has demonstrated that it is straightforward to develop a custom tool-based MetaRunner and it will encourage you to think about other reusable tools you could expose to TeamCity in the same way. For example, we wrap our Microsoft Azure Deployment Tooling in a MetaRunner to allow any endjineer (or customer) to deploy into Azure following our conventions by filling in a few simple fields relevant to their project in the TeamCity UI.

The source for both the SimpleRunner and PowerShellRunner versions of the plugin are available on GitHub.

Work Smarter, Not Harder.

@HowardvRooijen | @endjin

Endjin is a strategy, experience, development & cloud consultancy. Established in 2010, endjin has become the number one Microsoft Azure Partner in EMEA, delivering projects using cloud, mobile, desktop and embedded technologies in retail, financial services, and consumer applications. The company mantra is to “work smarter” and we help our customers do the same by improving their processes & engineering practices.

Subscribe to TeamCity Blog updates

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!