As you might have noticed, there was recently an option added to the Get TeamCity page of our website: AWS. This lets you run TeamCity in AWS using the official CloudFormation template.
In this post, we will go over what’s under the hood of the template, and why it may save you some time and effort.
Usually, installing TeamCity on top of AWS is quite a time-consuming task.
It requires the following steps:
- Setting up an external database,
- Configuring the EC2 instance to run a TeamCity server,
- Configuring it to then connect to the database,
- Installing the TeamCity server,
- Installing a TeamCity agent.
And then making the whole installation secure requires even more effort.
We have tried to ease this process and created an official CloudFormation template to run the TeamCity stack in AWS. Using this template lets you run all the above steps with just a single click. And should you decide to destroy the stack, CloudFormation also provides a super simple way to do it with just one click.
The template is located in the S3 bucket. The stack can be launched via the ‘Run on AWS’ button available on the TeamCity site.
The template provides several parameters:
It takes about 15 minutes for the template to deploy the whole stack, the most time-consuming task being the RDS Database instance roll up. Once the deployment is ready, you will see the TeamCity server endpoint in the Output section which points you to your TeamCity installation.
Just generate the root account and it’s ready to use.
So what is under the hood?
The TeamCity server runs on an EC2 instance with CoreOS Container Linux. The default agent runs as a separate container on the same instance. The external database is provided by an RDS MySQL instance. We decided not to introduce a custom AMI with TeamCity. Instead, we use the official Docker images with the TeamCity server and build agent.
The server and the database are placed in their own VPC which is completely secure. The DB allows only internal connections within the VPC. It’s only possible to connect to the Server via HTTP(s) or SSH.
How the server is running
There are several systemd services that prepare the LVM on the EBS volume to persist your data, create the file system, and run the latest official TeamCity Server and TeamCity Build Agent from the DockerHub images. Those services are linked to each other and roll the whole system back after an instance reboot or failure.
To connect to the server’s console, you need to use your instance key:
ssh -i instance-key.pem core@[server IP]
To see the logs, just run the docker logs command for the desired container.
Once you have TeamCity up and running, there are a few more steps to consider:
- Use the EC2 integration to run and connect build agents to your server, and
- Configure TeamCity to use the S3 bucket as external artifact storage.
Happy building with TeamCity on AWS!
Thanks, this is great! I was able to follow these instructions and get an instance up and running on AWS, I’m curious how an upgrade of this would be handled? say the next release after 2017.1.5? Is that documented somewhere already?
Fred, thank you for feedback!
Starting from 2017.2 you will be able to update TeamCity server from server admin UI. See more details here https://confluence.jetbrains.com/pages/viewpage.action?pageId=95391358#Indore2017.2EAP3(build49972)ReleaseNotes-ServerAutoUpgrade
As for 2017.1.x versions – we will publish document describing this scenario.
CloudFormation template seems good, but will it not be better if you could have leverage on AWS ECS, since you are already using docker images from docker hub?
Daniel, please provide more details how do you expect AWS ECS to be used?
By the way, there is a plugin to run build agents in ECS cluster https://github.com/JetBrains/teamcity-amazon-ecs-plugin
I think he means that rather than have an EC2 instance running the server and agent Docker images, you could use ECS to run them independently.
It would also be worth considering a Docker-based approach to agent provisioning per-build. Ephemeral agents, in my opinion, are critical to a reliable build pipeline.
Matt, thank you for clarification.
Actually EC2 instance run single (default) agent. The only purpose of it is to have a server with at least one agent connected from scratch.
In real life user should configure agents cloud with a help of one of related plugins:
–
https://confluence.jetbrains.com/display/TCD9/Setting+Up+TeamCity+for+Amazon+EC2
– https://plugins.jetbrains.com/plugin/10067-amazon-ecs-support
– https://blog.jetbrains.com/teamcity/2017/10/teamcity-kubernetes-support-plugin/
– etc.
By the way, all of those plugins support terminating agent after every build.
As for using power of ECS – we plan to provide HA (high available) TeamCity server installation in the future, so ECS will be used to run two instances of server. In current version usage of ECS looks like a big overhead.
Again, feel free to use ECS to run agents with ECS support plugin https://plugins.jetbrains.com/plugin/10067-amazon-ecs-support
I think automatic setting up of additional build agents is doable.
Docker CE for AWS supports setting up of a 1000-node cluster in a Cloudformation teamplte.
https://docs.docker.com/docker-for-aws/#docker-community-edition-ce-for-aws
It might be worth mentioning that there is ~/update-teamcity.sh which should probably be used to update from versions < 2017.2.
However I am now stuck with the following error and would appreciate any info:
[2017-11-28 11:27:48,661] ERROR – jetbrains.buildServer.STARTUP – Found a TeamCity internal database when no system directory or an empty system directory is expected
I'm using an auto-scaling group of 1 to make it easier to roll the server if required. When I terminate the instance the ASG created a new server and it booted up with this error. It upgraded ok and can see that it is now 2017.2, but I'd like to understand how to handle this error.
Matt, please submit an issue to https://youtrack.jetbrains.com/issues/TW
For future reference.
https://youtrack.jetbrains.com/issue/TW-52994
“Just generate the root account and it’s ready to use.”
I don’t know what I am missing here, but how do I do this step?
I don’t seem to be able to ssh onto the instance, and a quick look into the set up makes me think it doesn’t use my set up key value pair properly, which stops me from using ec2-user to ssh onto the box.
You should open TeamCity UI in browser. It will ask you to create TeamCity server root / admin account.
Thank you, thats great. That allows me to access the instance via browser, but how do I ssh to the instance box?
As mentioned in blog post you can do “ssh -i instance-key.pem core@[server IP]”
Is there anything remotely resembling an official terraform or cloudformation or definition to bring up a TeamCity server with MSBuild capability to compile a Solution file relying on Microsoft Specific APIs. I’m not talking about dotnet core, dotnet standard or . I have a legacy system to deal with, and need to bring up a CI environment. Looking at https://vimeo.com/157086224, the TeamCity + GoCD combo is what i need, but there does not seem to be any definition that makes the install process of TeamCity a snap if you have to work in a windows world. Cheers
Stephane, there are Windows based docker images for running TeamCity server or agent https://blog.jetbrains.com/teamcity/2017/10/teamcity-docker-images-for-windows/
As for Azure specific installation – we do not provide it at the moment. Please watch / vote for this issue https://youtrack.jetbrains.com/issue/TW-37094
Also you can always use any of cloud-based build agent providers to run Windows based agents.
Related to issue – https://youtrack.jetbrains.com/oauth?state=%2Fissue%2FTW-53860
Any recommendations for spinning up this template with no external IPs? I made some modifications to remove the EIP and got the deployment completed, however the web services don’t want to start without that IP apparently.
Chris, template currently doesn’t support this type of setup. We are working on it and will publish updated version. Please watch for news in https://youtrack.jetbrains.com/oauth?state=%2Fissue%2FTW-53860
How can I stop the default agent? When I stop the agent’s container it is automatically restarted, but it doesn’t seems to be configured as auto start in docker
Agent is being started as docker container via systemd service which is configured to always restart.
Currently it is not possible to stop agent. Please describe why do you need it? Is it critical issue for you?
We are using EC2 integration to spin up agents, we want to make sure all agents are aligned. This is the main reason. Also we want to make sure that the default agent won’t eat from TeamCity Server resources (as they are sharing the same instance).
10x!
Makes sence. Please watch for changes in this issue https://youtrack.jetbrains.com/issue/TW-53980
This would be perfect, but is there a version that doesn’t also start a build agent on the same instance? Ideally it would be great to just have the cloudformation to setup the server
Currently there is no such version. Please watch for changes in this issue https://youtrack.jetbrains.com/issue/TW-53980
Is there a possibility of adding a script to import TLS certificate to the Java Trust store which uses LDAP authentication . Currently I’ll have to manually get to the docker container to perform this operation .
Unfortunately no, but it make sence. I’ve created the issue https://youtrack.jetbrains.com/issue/TW-54334
access denied from:
https://s3.amazonaws.com/teamcity.jetbrains.com/teamcity-server.yaml
Fixed, thank you for the signal!
This runs and works fine for a short period (couple minutes to a couple days), but then it starts failing.
$ systemctl status oem-cloudinit
● oem-cloudinit.service – Cloudinit from EC2-style metadata
Loaded: loaded (/run/systemd/system/oem-cloudinit.service; static; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2018-04-11 21:53:14 UTC; 1min 30s ago
Process: 767 ExecStart=/usr/bin/coreos-cloudinit –oem=ec2-compat (code=exited, status=1/FAILURE)
Main PID: 767 (code=exited, status=1/FAILURE)
Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Wrote unit “teamcity-agent.service”
Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Ensuring runtime unit file “etcd.service” is unmasked
Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Ensuring runtime unit file “etcd2.service” is unmasked
Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Ensuring runtime unit file “fleet.service” is unmasked
Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Ensuring runtime unit file “locksmithd.service” is unmasked
Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Calling unit command “start” on “teamcity-server.service”
Apr 11 21:53:14 ip-10-0-0-180.ec2.internal coreos-cloudinit[767]: 2018/04/11 21:53:14 Failed to apply cloud-config: Unit format-mnt-data.service is not loaded properly: Exec format error.
Apr 11 21:53:14 ip-10-0-0-180.ec2.internal systemd[1]: oem-cloudinit.service: Main process exited, code=exited, status=1/FAILURE
Apr 11 21:53:14 ip-10-0-0-180.ec2.internal systemd[1]: oem-cloudinit.service: Failed with result ‘exit-code’.
Apr 11 21:53:14 ip-10-0-0-180.ec2.internal systemd[1]: Failed to start Cloudinit from EC2-style metadata.
$ systemctl status format-mnt-data
● format-mnt-data.service
Loaded: error (Reason: Exec format error)
Active: inactive (dead)
Apr 11 21:53:09 ip-10-0-0-180.ec2.internal systemd[1]: format-mnt-data.service: Cannot add dependency job, ignoring: Unit format-mnt-data.service is not loaded properly: Exec format error.
Apr 11 21:53:14 ip-10-0-0-180.ec2.internal systemd[1]: /etc/systemd/system/format-mnt-data.service:8: Failed to resolve unit specifiers on /usr/sbin/pvcreate /dev/xvdg && /usr/sbin/vgcreate app /dev/xvdg && /usr/sbin/lvcreate -l 100%FREE -n data app && /usr/sbin
Apr 11 21:53:14 ip-10-0-0-180.ec2.internal systemd[1]: /etc/systemd/system/format-mnt-data.service:8: Failed to resolve unit specifiers on /usr/sbin/pvcreate /dev/xvdg && /usr/sbin/vgcreate app /dev/xvdg && /usr/sbin/lvcreate -l 100%FREE -n data app && /usr/sbin
Got same problem.
SystemD uses % as the keyword for internal format specifiers. In UserData section 100%FREE should be escaped: 100%%FREE.
Andrew, thanks, the template with a fix was deployed and it is ready to use.
I forgot to note that I use all default settings except for KeyName and DBPassword. When I create the stack, it takes about 15 minutes, then I go to the provided IP and am able to agree to terms, create a user, etc and start configuring and using it to do builds but then at some point it just fails to connect. So I then ssh to the instance and that is when I output the logs as posted earlier. I have had this happen multiple times.