Amazon SNS Trigger Plugin Announcement

About

We’ve created a new TeamCity plugin that brings the functionality of triggering builds by new messages into an Amazon Simple Notification Service (Amazon SNS).

Example

Intro

Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.

Source

Amazon SNS is one of the backbone AWS services that allows connecting various applications and services to each other. The typical use case for Amazon SNS is to provide notifications about events in one system and trigger corresponding actions in others.

We’ve set up a data processing pipeline so that you can see it in action. We’ll build a pipeline that takes a newly uploaded image, optimizes it for the web, and creates a few thumbnails of different sizes.

Our pipeline begins by placing the new image file in an S3 bucket. TeamCity is responsible for running the build configuration. The challenge is making sure that the new file is detected when it appears. The solution is simple. As with a significant part of Amazon cloud services, AWS S3 integrates with Amazon SNS out of the box with its “Publisher” role. This is a natural way to notify external systems about new files in buckets. TeamCity takes the “Subscriber” role.

The setup steps look like this:

• On the AWS side:

1. Create an AWS S3 Bucket with a name, e.g., ‘demo-storage-image-processing-pipeline’.
2. Create an AWS SNS Topic with a name, e.g., ‘new-image-into-the-demo-storage’.
3. Set the bucket to push notifications to the topic.
4. Create an IAM user for TeamCity and attach the ‘AmazonS3FullAccess’ IAM policy.
5. Create access and secret access keys.

• On the TeamCity side:

1. Create a build configuration.
2. Set up the trigger.
3. Set up an AWS Connection using the IAM user’s key.
4. Grant the build configuration permission to interact with AWS.
5. Create the pipeline steps.

Firstly, we need to create resources on the AWS side. How to create an SNS topic and S3 bucket is out of the scope of this article, but you can find step-by-step instructions in the official documentation.

Build configuration

Now it’s time to create a build configuration with the trigger that will run our processing tasks. TeamCity will ask us about a VCS root we want to use, but since we don’t need one, we can skip this.

In General Settings, specify the Artifact path. We’ll use it to demonstrate a result.

Now we have an empty build configuration. Before we start working on our steps, we need to do a couple of things:

• Extract image properties like its name and the storge bucket from the SNS Trigger 
• Grant TeamCity permission to get files from the bucket.

Trigger

When a SNS message will trigger our process, TeamCity puts the message into a parameter used in later steps. Let’s create the parameter that we will use – the value can be anything. The trigger will update the value for every running build as every new message appears.

On the Triggers page, add a new Amazon SNS Trigger. Specify its name, and don’t forget to copy the endpoint URL.

On the Build Customization tab, ask TeamCity to use the message parameter to publish an SNS message.

Don’t forget to save the trigger!

Now go to the AWS console, the SNS service, and subscribe to TeamCity by using the provided endpoint URL.

That’s it! You can check the trigger in TeamCity to see if it was updated correspondingly in the Subscription Info section:

AWS Connection

Now it’s time to grant the scripts permission to download images from the S3 bucket. On the project level, add a new connection for AWS using the same IAM user credentials we created before.

Check the connection using the Test Connection button:

Next, we want to check our build steps will work with AWS. Let’s return to our build configuration. On the Build features page using the “AWS Credentials” build feature, TeamCity will now securely provide credentials for the build steps.

Build steps

Now we are ready to set up our build steps!

In this demo, we will use the Command line runner for our build steps.

Build step #1: Get the information from the message

The custom script is:

# Extract bucket name and image name from the JSON message
BUCKET=$(echo %message% | jq -r '.Records[].s3.bucket.name')
IMAGE=$(echo %message% | jq -r '.Records[].s3.object.key')

# Ask TeamCity to remember the variables to use them 
# in the next steps
echo "##teamcity[setParameter name='env.BUCKET' value='${BUCKET}']"
echo "##teamcity[setParameter name='env.IMAGE' value='${IMAGE}']"

We will use the Docker wrapper feature to run the script inside the Docker container based on the official jq Docker image. Set ‘Run step within Docker container’ to the ‘stedolan/jq’ value. That allows us to use the jq utility on any agent, even if the agent does not have the utility preinstalled.

Build step #2: Download the image from the S3 bucket

We need to download the file from AWS S3 using AWS CLI. Again, the Docker wrapper will help us get the latest version of the AWS CLI utility.

Custom script:

aws s3 cp s3://${BUCKET}/${IMAGE} image.png

Set Run step within Docker container to the ‘amazon/aws-cli’ value.

Build step #3: Optimize image and create thumbnail

For this step, we are going to use two utilities: OptiPNG to optimize the image file size and the convert utility from the ImageMagick CLI graphical package. Unfortunately, the vendors didn’t provide us with official Docker images, so we’ll install them at run time.

Again we’ll use the Docker wrapper, but this time we use a basic ‘Ubuntu’ Docker image and install the required packages during the execution of this step. This will give us the required binaries, and we won’t litter inside the agent.

All installed packages will be erased with the container together. This approach is helpful while you construct your pipeline, but avoid it in production! When you learn all of the required dependencies, build your own Docker image with all of the packages preinstalled. It will save traffic and agent time.

Custom script:

apt -qq update &>/dev/null 
apt -qq install optipng imagemagick -y  &> /dev/null

# optimize our image
cp image.png optimized_image.png
optipng optimized_image.png

# create a thumbnail with the 200x400 size
convert optimized_image.png -strip -thumbnail 200x400 thumbnail.png

Run it!

Let’s put some .png files into our bucket using the AWS console. A few moments later, a new build will be scheduled and run. As a result, the three images are now available as build artifacts.

Of course, this is just an example; you can create much more complex pipelines. Triggers can be useful for various event-based automation tasks.

How to get it

The plugin is available in JetBrains Marketplace. Don’t hesitate to give us feedback here – we always appreciate your opinion! Our development team will be delighted to hear how you feel about the plugin.

Conclusion

With Amazon SNS integration, TeamCity can be built into multiple scenarios. It can orchestrate tasks in different cloud services based on the event trigger model. Here’s a list of AWS services that can publish messages to Amazon SNS. With the support of custom integrations, the scope of possibilities is enormous. Please share the ways you use TeamCity with Amazon SNS!