The official TeamCity CloudFormation template

As you might have noticed, there was recently an option added to the Get TeamCity page of our website: AWS. This lets you run TeamCity in AWS using the official CloudFormation template.

get-teamcity-aws

In this post, we will go over what’s under the hood of the template, and why it may save you some time and effort.

Usually, installing TeamCity on top of AWS is quite a time-consuming task.
It requires the following steps:

  • Setting up an external database,
  • Configuring the EC2 instance to run a TeamCity server,
  • Configuring it to then connect to the database,
  • Installing the TeamCity server,
  • Installing a TeamCity agent.

And then making the whole installation secure requires even more effort.

We have tried to ease this process and created an official CloudFormation template to run the TeamCity stack in AWS. Using this template lets you run all the above steps with just a single click. And should you decide to destroy the stack, CloudFormation also provides a super simple way to do it with just one click.

The template is located in the S3 bucket. The stack can be launched via the ‘Run on AWS’ button available on the TeamCity site.

The template provides several parameters:

aws-1
It takes about 15 minutes for the template to deploy the whole stack, the most time-consuming task being the RDS Database instance roll up. Once the deployment is ready, you will see the TeamCity server endpoint in the Output section which points you to your TeamCity installation.

aws-2

Just generate the root account and it’s ready to use.

So what is under the hood?

The TeamCity server runs on an EC2 instance with CoreOS Container Linux. The default agent runs as a separate container on the same instance. The external database is provided by an RDS MySQL instance. We decided not to introduce a custom AMI with TeamCity. Instead, we use the official Docker images with the TeamCity server and build agent.

The server and the database are placed in their own VPC which is completely secure. The DB allows only internal connections within the VPC. It’s only possible to connect to the Server via HTTP(s) or SSH.

How the server is running

There are several systemd services that prepare the LVM on the EBS volume to persist your data, create the file system, and run the latest official TeamCity Server and TeamCity Build Agent from the DockerHub images. Those services are linked to each other and roll the whole system back after an instance reboot or failure.

To connect to the server’s console, you need to use your instance key:
ssh -i instance-key.pem core@[server IP]
To see the logs, just run the docker logs command for the desired container.

Once you have TeamCity up and running, there are a few more steps to consider:

Happy building with TeamCity on AWS!

This entry was posted in Features, FYI, How-To's. Bookmark the permalink.

32 Responses to The official TeamCity CloudFormation template

  1. Fred Vaughn says:

    Thanks, this is great! I was able to follow these instructions and get an instance up and running on AWS, I’m curious how an upgrade of this would be handled? say the next release after 2017.1.5? Is that documented somewhere already?

  2. Daniel Leong says:

    CloudFormation template seems good, but will it not be better if you could have leverage on AWS ECS, since you are already using docker images from docker hub?

  3. Matt Canty says:

    It might be worth mentioning that there is ~/update-teamcity.sh which should probably be used to update from versions < 2017.2.

    However I am now stuck with the following error and would appreciate any info:

    [2017-11-28 11:27:48,661] ERROR – jetbrains.buildServer.STARTUP – Found a TeamCity internal database when no system directory or an empty system directory is expected

    I'm using an auto-scaling group of 1 to make it easier to roll the server if required. When I terminate the instance the ASG created a new server and it booted up with this error. It upgraded ok and can see that it is now 2017.2, but I'd like to understand how to handle this error.

  4. David da Silva says:

    “Just generate the root account and it’s ready to use.”
    I don’t know what I am missing here, but how do I do this step?

    I don’t seem to be able to ssh onto the instance, and a quick look into the set up makes me think it doesn’t use my set up key value pair properly, which stops me from using ec2-user to ssh onto the box.

    • Evgeniy Koshkin says:

      You should open TeamCity UI in browser. It will ask you to create TeamCity server root / admin account.

      • David da Silva says:

        Thank you, thats great. That allows me to access the instance via browser, but how do I ssh to the instance box?

  5. Stephane Mikaty says:

    Is there anything remotely resembling an official terraform or cloudformation or definition to bring up a TeamCity server with MSBuild capability to compile a Solution file relying on Microsoft Specific APIs. I’m not talking about dotnet core, dotnet standard or . I have a legacy system to deal with, and need to bring up a CI environment. Looking at https://vimeo.com/157086224, the TeamCity + GoCD combo is what i need, but there does not seem to be any definition that makes the install process of TeamCity a snap if you have to work in a windows world. Cheers

  6. Chris Murdock - Conga says:

    Related to issue – https://youtrack.jetbrains.com/oauth?state=%2Fissue%2FTW-53860

    Any recommendations for spinning up this template with no external IPs? I made some modifications to remove the EIP and got the deployment completed, however the web services don’t want to start without that IP apparently.

  7. Alex says:

    How can I stop the default agent? When I stop the agent’s container it is automatically restarted, but it doesn’t seems to be configured as auto start in docker

    • Evgeniy Koshkin says:

      Agent is being started as docker container via systemd service which is configured to always restart.
      Currently it is not possible to stop agent. Please describe why do you need it? Is it critical issue for you?

  8. Kevin Baxtrom says:

    This would be perfect, but is there a version that doesn’t also start a build agent on the same instance? Ideally it would be great to just have the cloudformation to setup the server

  9. Is there a possibility of adding a script to import TLS certificate to the Java Trust store which uses LDAP authentication . Currently I’ll have to manually get to the docker container to perform this operation .

  10. Josh Coady says:

    This runs and works fine for a short period (couple minutes to a couple days), but then it starts failing.

    $ systemctl status oem-cloudinit
    ● oem-cloudinit.service – Cloudinit from EC2-style metadata
    Loaded: loaded (/run/systemd/system/oem-cloudinit.service; static; vendor preset: disabled)
    Active: failed (Result: exit-code) since Wed 2018-04-11 21:53:14 UTC; 1min 30s ago
    Process: 767 ExecStart=/usr/bin/coreos-cloudinit –oem=ec2-compat (code=exited, status=1/FAILURE)
    Main PID: 767 (code=exited, status=1/FAILURE)

    Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Wrote unit “teamcity-agent.service”
    Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Ensuring runtime unit file “etcd.service” is unmasked
    Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Ensuring runtime unit file “etcd2.service” is unmasked
    Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Ensuring runtime unit file “fleet.service” is unmasked
    Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Ensuring runtime unit file “locksmithd.service” is unmasked
    Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Calling unit command “start” on “teamcity-server.service”
    Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Failed to apply cloud-config: Unit format-mnt-data.service is not loaded properly: Exec format error.
    Apr 11 21:53:14 ip-10-0-0-180.ec2.internal systemd[1]: oem-cloudinit.service: Main process exited, code=exited, status=1/FAILURE
    Apr 11 21:53:14 ip-10-0-0-180.ec2.internal systemd[1]: oem-cloudinit.service: Failed with result ‘exit-code’.
    Apr 11 21:53:14 ip-10-0-0-180.ec2.internal systemd[1]: Failed to start Cloudinit from EC2-style metadata.

    $ systemctl status format-mnt-data
    ● format-mnt-data.service
    Loaded: error (Reason: Exec format error)
    Active: inactive (dead)

    Apr 11 21:53:09 ip-10-0-0-180.ec2.internal systemd[1]: format-mnt-data.service: Cannot add dependency job, ignoring: Unit format-mnt-data.service is not loaded properly: Exec format error.
    Apr 11 21:53:14 ip-10-0-0-180.ec2.internal systemd[1]: /etc/systemd/system/format-mnt-data.service:8: Failed to resolve unit specifiers on /usr/sbin/pvcreate /dev/xvdg && /usr/sbin/vgcreate app /dev/xvdg && /usr/sbin/lvcreate -l 100%FREE -n data app && /usr/sbin
    Apr 11 21:53:14 ip-10-0-0-180.ec2.internal systemd[1]: /etc/systemd/system/format-mnt-data.service:8: Failed to resolve unit specifiers on /usr/sbin/pvcreate /dev/xvdg && /usr/sbin/vgcreate app /dev/xvdg && /usr/sbin/lvcreate -l 100%FREE -n data app && /usr/sbin

    • Andrew says:

      Got same problem.
      SystemD uses % as the keyword for internal format specifiers. In UserData section 100%FREE should be escaped: 100%%FREE.

      • Dmitry Tretyakov says:

        Andrew, thanks, the template with a fix was deployed and it is ready to use.

  11. Josh Coady says:

    I forgot to note that I use all default settings except for KeyName and DBPassword. When I create the stack, it takes about 15 minutes, then I go to the provided IP and am able to agree to terms, create a user, etc and start configuring and using it to do builds but then at some point it just fails to connect. So I then ssh to the instance and that is when I output the logs as posted earlier. I have had this happen multiple times.

Comments are closed.