Continuous delivery for databases brings speed, efficiency and predictability to your release cycle, by automating database deployments across a pipeline. Setting-up version control, continuous integration (CI) and automated release management can provide you with a steadier stream of reliable releases.
In this practical demo, Alex Yates, Pre-Sales Engineer at Red Gate, will show you how to set up continuous delivery for your SQL databases using TeamCity, Red Gate’s SQL Automation Pack, and Octopus Deploy. You’ll learn best practices for continuous integration, continuous delivery and automated deployments to set-up your own pipeline.
Space is limited, please register now. There will be an opportunity to ask questions during the webinar.
About the Presenter:
Alex Yates has worked with database change management tools for four years, collaborating closely with users and dev teams along the way. As a pre-sales engineer, he gets to see a huge variety of server and dev environments, and helps folks solve their database development and delivery problems in whatever way works well for them. Ever the sharer, he also blogs about the lessons he learns: www.workingwithdevs.com.
This guest blog post is from Wes McClure, Software Consultant and Founder of Full City Tech Co., a JetBrains Training and Consulting Partner.
In Approaching Automation, I outlined a series of steps to follow to make worthwhile investments in automation. In the following example, I’ll show how to apply these steps to a software development process.
Inspecting code can provide valuable insight into improving the design of a system. Inspection tools are most valuable when they’re integrated with the environments that developers use to create software. They can provide instant feedback to improve, on the fly.
Inspection is also valuable as an analysis tool after the fact. One barrier to analysis is the time it takes to setup and inspect a code base. This process is ripe for automation so time can be spent analyzing results, not gathering them.
Wes McClure, Software Consultant and Founder of Full City Tech Co.
But, blindly automating anything is as reckless as avoiding automation altogether.
Often, when discussing inspections, I find individuals wanting to inspect code bases every time a change is made to the system.
I’ll inquire how often do you perform this analysis currently and what do you do with the information? Often, I find there’s no methodical approach and sometimes inspection isn’t even a part of an existing process. It’s just something that someone said was a good idea.
Whatever the case, by stepping back and discussing how often the information is used and what it’s used for, we can begin to understand the value of automating inspections.
By challenging ourselves to understand why the information is valuable, we can determine the appropriate level of automation to improve.
Most teams are busy enough, they’ll be lucky to look at inspection results once a week. And, if they have to manually generate it, it’s much less likely they’ll even get around to it.
But, if inspection reports are automatically available on a weekly basis, they could invest more time in analyzing the results. And in turn, invest in acting on the results.
NDepend is a tool to inspect a .NET code base and provide actionable metrics to improve.
TeamCity is a platform to automate development processes, gather results and act upon them.
Let’s walk through the basis for automating inspections with NDepend and TeamCity.
First, we should outline the process.
NDepend is used to inspect a code base. This requires checking out the code, compiling it and then analyzing the results with NDepend.
Then, teams analyze the results for ways to improve.
And over time, they apply this insight to incrementally improve.
Eliminate the unnecessary
After outlining the process, it’s important to eliminate vestigial components. In the case of inspections, this is a matter of making a conscientious decision about what inspections are meaningful and what aren’t. Don’t just take everything out of the box. And, spend some time with NDepend to craft your own custom inspections.
Next, it’s important to establish objectives. NDepend comes with the concept of rules. Reducing rule violations can improve the quality of a code base. For example, NDepend comes out of the box with rules that help detect breaking changes to software interfaces. It also provides rules to detect dead code which can hamper the longevity of software. Deciding on a set of rules to enforce may serve as a worthwhile objective.
Let’s say we want to reduce dead code in a system. Every system contains some amount of dead code. Some of it can be detected automatically. Measuring the current level of dead code in a system and setting goals to reduce it serves as a progress indicator.
What if your system is comprised of ten percent dead code? What would it be worth the get that to five percent? What about one percent?
Make a decision
Everything above becomes the basis with which one decides to automate, or not automate, inspections of dead code or any other aspect of a system.
I always recommend a margin of two or three times the potential cost. That way you have room to absorb the unknown.
There’s no better way to describe automating it than to show you:
Over time, use the information you capture in TeamCity from the output of NDepend to see if efforts prove worthwhile.
Not everything is so easily quantifiable. Nonetheless, you can start to see how you can apply a methodical process focused on value to scientifically improve your development process.
The original post was published September 11th, 2014 on www.wesmcclure.com.
We are researching the ecosystem of the CI tools that you are using and we’ll appreciate if you could help us with this. You surely have got plenty of expert knowledge when it comes to using different CI tools and we’ll be happy if you shared your opinion and experience with us by spending 10-15 minutes on completing our survey. Of course, we hope that TeamCity is your favorite tool, but even if it’s not, we would like you to participate in the survey and help us understand what we could do to make TeamCity better.
Our survey is designed to be completed in one sitting, which means that once you close it, you won’t be able to go back to the questions you’ve already answered. Ready, set, go! We guarantee that all the information you provide will be kept confidential.
In case you think you’ve got better things to do, think twice! There is a bonus – those who complete the survey will be eligible for one of three $300 Amazon.com certificates. Don’t wait for long – get started with the survey!
The prize winners will be named on October, 10. Check your email to find out if you are one of them! There will also be an announcement in our blog and Twitter!
Stay tuned for the latest TeamCity news and happy building!
For quite some time now, before every new version, we’ve been running a very successful Early Access Program (EAP) which helps us improve TeamCity basing on feedback from our most enthusiastic users. So today we are happy to announce that the Early Access Program for TeamCity 9.0 is now officially open! Traditionally TeamCity versions are code-named after Indian cities, and this one is no exception, so please welcome Hajipur 9.0 EAP1 (build 31423).
TeamCity 9.0 plans
Before I talk about the features of this release, let me briefly outline the main directions for TeamCity 9.0.
The estimated release date for the new version is October 2014, which will probably be corrected as we approach the release.
One of the major directions is the ability to move projects among servers. We are currently working on ensuring safe and foolproof transfer of projects.
Among the priorities of the future version is the functionality allowing TeamCity users to store the project configuration settings in the VCS the same way it is done with the source code.
The advent of the cloud solutions gives one more significant course to our work: our efforts are directed toward cloud integration with VMware Vsphere and Microsoft Azure.
In addition to developing these new features, we are also investing considerable resources into improving TeamCity performance, reliability and ease of use.
Highlights of Hajipur 9.0 EAP1 (build 31423)
This release contains a snapshot of our continuing work on the upcoming release. We’ve been working hard and today we are presenting: two-way synchronization of your project settings with the version control – now you can store the project configuration settings in the VCS the same way it is done with the source code, Build Time Report, Unicode support for MS SQL and Oracle databases used by TeamCity and other features, such as the ability to upload meta-runners, tagging queued and running builds and more. We also introduced some changes aimed at speeding up the backup restore operations as well as clean-up, which is gradually being moved to the background.
We encourage everyone to refer to our release notes for details and, if you want to be a part of the TeamCity development process, download this EAP version and give it a try! We are collecting your feedback, so keep it coming! We are thrilled to hear what you have to say!
Help us make the new TeamCity better and happy building!
I enviously hope the summer is treating you well and you are enjoying the bright sun with an ice-cold drink in hand while writing code on your laptop in the atmosphere of complete harmony and relaxation!
Why am I fantasizing that the grass is greener on your side of the fence? That’s because this summer is blazing hot for the TeamCity development team, but we are craving cold drinks: trying to set right any smallest defect of TeamCity 8.1 while simultaneously brewing the brand new TeamCity 9.0!
So, today we are rolling out a new update, TeamCity 8.1.4, with over 50 fixes and improvements in various areas; the notable ones are several performance fixes, the fix for the regression of database restore performance which surfaced in 8.1.3, and the problem with NuGet Feed authentication. As you may have noticed, TeamCity is getting more and more stable: the number of issues reported and fixed in build 30168 is considerably smaller than our usual 100+.
What else? The bundled DotCover version has been updated to 2.7, and there is more: check out our Release Notes for the complete list of the bits of awesomeness brought by this version!
Build 30168 uses the same database version as all TeamCity 8.1.x releases, so why not give it a try when you can downgrade at any moment to one of 8.1.x builds?
Today I want you to take a peek behind the scenes of the TeamCity development process. Some time ago our team of superstars held the TeamCity plugin week which has by now become traditional. The period in-between releases seemed to be the perfect time for our developers to unleash their creativity and boy, they had a ball!
We hope to arouse your interest, but be warned: some of the plugins are pretty raw, in pre-alpha state, and PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND. Do not install them in production without testing!
Being given time to put their thoughts into lines of code, our developers went really wild, and came up with a number of ideas! Prepare yourselves for the unexpected!
We have a lot of votes for Sonar integration in our tracker. So please welcome the initial version of the Sonar plugin! The plugin comes with the Sonar build step and a simple user interface for configuring Sonar servers. More will be added based on your feedback!
A sense of humor is a sign of intelligence, and this plugin is indispensable for TeamCity as an intelligent Continuous Integration server! It’s all about challenging and motivating your developers: the plugin grants achievements to users for record numbers of various actions in the TeamCity web interface.
You will be bestowed the proud name of a Saboteur for bombing (i.e. muting) several tests with automatic explosion (unmute) on a specific date
Those disarming (unmuting) several tests, will be called brave Sappers
You’ll see how many Novelists you have with this achievement granted to the code contributors for extra-long commit descriptions.
The recording of our June 26th webinar with Wes McClure, TeamCity: Beyond Continuous Integration, is now available on JetBrains YouTube Channel.
In this webinar, we explore the landscape of opportunities beyond Continuous Integration. See how build chains and dependencies provide a robust framework to orchestrate the benefits automation can bring, to the software you create.
Continuous Integration is a gateway to further automation. It lays a foundation to change software quickly, with confidence. But why stop there? As we experiment with automation, we can establish techniques to further test, distribute and deliver software. Can you imagine what it would be like to focus on adding valuable features without being burdened with how they’re released to the world?
Come experiment with us as we explore the landscape of opportunities beyond Continuous Integration. See how build chains and dependencies provide a robust framework to orchestrate the benefits automation can bring, to the software you create.
Space is limited, please register now. There will be an opportunity to ask questions during the webinar.
About the Presenter:
Wes McClure is passionate about helping companies achieve remarkable results with technology and software. He’s had extensive experience developing software and working with teams to improve how software is developed to meet business objectives. Wes launched Full City Tech to leverage his expertise to help companies rapidly deliver high quality software to delight customers. He has a strong background in using Continuous Integration with TeamCity to bring quality to the table.
This guest blog post is brought to you by Howard van Rooijen of endjin, a JetBrains Training and Consulting Partner. Your feedback is welcome in the comments section.
As systems grow in size and complexity, without rigorous Application Lifecycle Management processes in place, organisations risk degrading into a state of chaos and fear. Earlier in the year I wrote a whitepaper for JetBrains, “From Chaos, through Fear, to Confidence” which describes how endjin helped one of our customers regain their confidence in their development process through the introduction of engineering practices, automated processes and feedback loops that have become the main tenets of the DevOps movement:
It is very rare to find a “one size fits all” solution for this type of process automation, but a more pragmatic approach of picking the “best of breed” tooling in each area and stitching them together into a cohesive solution can yield great results. Integration and extensibility points become key; luckily TeamCity and YouTrack have plenty of both.
In the previous diagram there is one important feedback loop missing:
We’re not notifying NewRelic that a new deployment has occurred. One of the main concerns of operational monitoring is checking to see that a change to any production system doesn’t negatively impact performance or reliability, and the most likely time this will happen is when a deployment has occurred.
For those of you who are unfamiliar with NewRelic, it’s a cloud based application monitoring platform that allows you to keep tabs on the CPU & memory utilisation, number of web transactions, unhandled errors and user-definable transactions within your application. We have used NewRelic on a number of our Microsoft Azure projects, from the development environment, through test and UAT and production. It’s an invaluable tool gaining application telemetry if you deploy into a cloud environment, where you don’t really have access to the operating system.
NewRelic has a concept of a deployment as an application event, but requires this to be raised by a 3rd party. By default the charts are unadorned by deployment information and the deployment event stream is empty:
This guide describes how to take a pre-existing tool, wrap it in a tool-based MetaRunner with a custom user interface, which can be used by TeamCity to notify NewRelic that a deployment has occurred:
Once NewRelic is notified a deployment has happened, new diagnostic features are enabled to help analyse deployment related performance changes.
In essence, a MetaRunner is a user-definable set of build steps, which can either be written from scratch by hand or extracted from existing, configured build steps, which you can parameterize and then create a simple user interface over, using primitive controls (textbox, checkbox, dropdown list) and validation, to enable easy re-use. In this case a MetaRunner is simply an xml file, which contains the definition of the build steps and also an embedded version of scripts you wish to execute.
MetaRunners can be used in another (as yet, undocumented) way; by following a series of conventions, you can package up tools and scripts, create a custom UI definition to allow easy reuse, package them up as a zip and deploy on to your TeamCity server. This type of tool-based MetaRunner plugin will become available as a build runner via the TeamCity UI.
1. Creating the tool-based MetaRunner Plugin
The structure for a MetaRunner plugin is as follows:
At the root level of the plugin there is a teamcity-plugin.xml file. This contains all the information that will be displayed in the TeamCity Build Runner UI (i.e. name, description etc.)
The second teamcity-plugin.xml located file under the agent directory is simpler – it is just a simple descriptor which signifies that the plugin is a tool package (i.e. it contains binaries and scripts you wish to include with the MetaRunner):
All of the tooling files you wish to include should also be stored under the agent directory (I have created a bin sub-folder, in this case it’s not really needed, but if you want to package up lots of tools / files, it’s simpler to keep the tools separate from the mandatory plugin files).
The final required file called MRPLUGIN_NewRelicDeploymentNotifier.xml and is located in server/metaRunners:
This file is quite complex at first glance, but is actually quite trivial. The document contains two main sections; parameters, which defined the parameters you allow users to configure via the UI, that will be passed into the context of the MetaRunner upon execution and build-runners which described the build runner you want to execute (the two most straight forward are; ‘simpleRunner’ which equates to running within the shell environment and ‘jetbrains_powershell’ which allows you to run a PowerShellScript). Note how we just reuse the text from the NewRelic Deployment Notification documentation for the description fields for the UI. In this example we are using a simpleRunner to wrap the required Curl command from the NewRelic ReST API inside the script.content element:
As you can see, the parameters we defined in the parameters section can be referenced inside the script.content element by using %your.custom.parameter%. One gotcha is to ensure that you need to wrap the path to the tool and its parameters with " characters to ensure paths and values are resolved correctly. One important item to note is the path to the tool .exe:
%teamcity.agent.tools.dir% is a standard TeamCity environment variable, but nr-deployment-notifier is derived from the name of the tool package zip file you create from the agent directory (see steps below).
2. Packaging the tool-based MetaRunner Plugin
Now that you have all the configuration files and the tools you want to include, you need to package them up in a zip file. Firstly, you need to package up the contents of the agents folder, as this zip file will be distributed to all build agents by TeamCity. I originally wrote a build script which automated this process, but there seems to be some incompatibility between 7Zip and the Java APIs that TeamCity uses to unpackage the zip file, so you need to resort to using the built in zip features of Windows:
As mentioned above, the name you give this zip file is important as the contents will be unpackaged into a folder of the same name, which you need to refer to in your script.content element:
You can delete the rest of the files, so just the zip is located in the agents directory (obviously, do this to a copy of your files, not the originals):
Next you need to package the whole plugin directory the same way:
The name of this file doesn’t have any repercussions, so any name will do:
3. Installing the MetaRunner Plugin
Next you need to install the zip into the main TeamCity plugins directory, located at <TeamCity Data Directory>/plugins. The screenshot below shows a single server TeamCity environment (where both the Build Agent and TeamCity Server are located on the same OS). The left hand windows shows the Build Agent Tools folder located at <TeamCity Home>/buildAgent/tools (where the MetaRunner package will be automatically unpacked) and on the right hand side the <TeamCity Data Directory>/plugins folder.
Once you have dropped the plugin into the <TeamCity Data Directory>/plugins folder, after a few minutes a new folder, with the same name as the zip you created inside the agents directory, should appear. If the new directory doesn’t appear after two minutes, you may need to restart the Build Agent service.
4. Using the MetaRunner Plugin
Now that the plugin has been successfully deployed to the TeamCity server, you should be able to create a new build step and select the “NewRelic Deployment Notifier” as the Runner type. You should now be able to see the rest of the UI you have defined:
The two mandatory fields are the API Key and the App Name of the app that has been deployed. The remaining fields are optional, but you can add the standard TeamCity environment variables %build.number% to automatically pass in the build number of the deployment that’s occurred and %teamcity.build.triggeredBy% to pass in the name of the user who kicked off the deployment.
If the build step runs successfully, you should see the following output:
Side Note: in order to interact with HTTPS services, Curl has a dependency on OpenSSL. In order to use this MetaRunner, you will need to install OpenSSL on each of your Build Agents. If you do not have this installed, the above build step with fail with an error message like: Process exited with code –1073741515. More information about how I diagnosed this requirement in the “Troubleshooting the MetaRunner Plugin” section below.
The end result
With the deployment notifier properly configured, every time a deployment is carried out by TeamCity, NewRelic with visualize the deployment as a vertical line across its graphs. If you hover over the line with your mouse, the details you passed into the notifier are displayed:
Once you have notified NewRelic of a deployment, some new features appear, such as the ability to analyze performance since the last deployment:
Troubleshooting the MetaRunner Plugin
Once I deployed the original SimpleRunner version of the MetaRunner plugin, I saw the following “Process exited with code –1073741515″ error. To try and diagnose the issue, I wanted to peek into the temporary .cmd file (highlighted in yellow) that TeamCity had generated to ensure that my script was properly written, but unfortunately, the file is deleted as soon as it has been run.
So I created a new PowerShellRunner version of the MetaRunner plugin, that allowed me to add more validation and debugging logic:
This all seemed correct, but unlike the SimpleRunner, when I navigated to the location highlighted in yellow, the file still existed, so I opened it in PowerShell ISE and ran it and saw a Windows exception dialog with the following message from Curl:
The program can't start because libsasl.dll is missing from your computer.
Try reinstalling the program to fix this problem.
Installing OpenSSL solved the problem, but I thought the fact that the .PS1 script does not appear to get deleted by TeamCity 8.x could be a useful technique for anyone else trying to diagnose why their MetaRunner is failing.
Hopefully this guide has demonstrated that it is straightforward to develop a custom tool-based MetaRunner and it will encourage you to think about other reusable tools you could expose to TeamCity in the same way. For example, we wrap our Microsoft Azure Deployment Tooling in a MetaRunner to allow any endjineer (or customer) to deploy into Azure following our conventions by filling in a few simple fields relevant to their project in the TeamCity UI.
The source for both the SimpleRunner and PowerShellRunner versions of the plugin are available on GitHub.
Endjin is a strategy, experience, development & cloud consultancy. Established in 2010, endjin has become the number one Microsoft Azure Partner in EMEA, delivering projects using cloud, mobile, desktop and embedded technologies in retail, financial services, and consumer applications. The company mantra is to “work smarter” and we help our customers do the same by improving their processes & engineering practices.
In this webinar we use dotCover to collect code coverage information while running tests in our CI process. We can see how we can configure code coverage and how we can use the TeamCity Visual Studio plugin to download the coverage snapshot generated on the build server and inspect it using dotCover on a developer machine.
How much of our code is being covered by our unit tests? Are there areas we are not testing? By capturing code coverage data during a test run, we can analyze which areas of our applications are well-tested and which ones require additional tests to be written. And where better to capture code coverage information than on our build server? Find out in this webinar.
Below are select questions and answers from our webinar:
Q: Can we use MStest with TeamCity and also get code coverage with dotCover?
A: Yes, test runner type and .NET coverage tool can be chosen in TeamCity Build Configuration Settings tab. Please check the following settings:
Q: Is it possible to use dotCover on TeamCity with xUnit.net tests?
A: Yes, it is possible. If you are using xUnit and you want to capture code coverage, you can run dotCover from the command line and import the data into TeamCity by using a service message. The process details are described by Maarten during the webinar (starting from 24th minute).
Maarten Balliauw is a Technical Evangelist at JetBrains. His interests are all web: ASP.NET MVC, PHP and Windows Azure. He’s a Microsoft Most Valuable Professional (MVP) for Windows Azure and an ASPInsider. He has published many articles in both PHP and .NET literature such as MSDN magazine and PHP architect. Maarten is a frequent speaker at various national and international events such as MIX (Las Vegas), TechDays, DPC and others.