Investigating a Compromised TeamCity On-Premises Server and Its Build Environment
Introduction
If you suspect your TeamCity On-Premises server has been compromised through a security vulnerability (such as CVE-2024-27198), and you were unable to patch or upgrade your server prior to the suspected incident, this blog post recommends some of the steps you should take to investigate the incident, along with potential actions for remediation.
You can also contact the TeamCity Support team if you have any additional questions or concerns.
Areas of investigation
Server
As a first step, your TeamCity server should be isolated from both internal and external networks. If the server has been compromised, there may be unknown processes running on it that could still access resources on the local machine and broader internal network. It’s also possible that an attacker may have established a backdoor communication channel.
Keep in mind that, if the server has been compromised, any illicit actions would be performed through the user account configured for the TeamCity server process. This user account may have privileged access to other resources on the machine, or in the case of a domain user account, additional resources on your local network.
Build agents
Build agents connected to the TeamCity server at any point while it was compromised may be considered at risk if a modified build was executed on an agent, a plugin was installed, or the Agent Terminal feature was used to connect directly to an agent and run commands. The audit logs (described below) will highlight any such cases.
If you suspect that any build agents have been compromised, they should be isolated from any internal and external networks. You should consider restoring them to a previous known state (prior to their becoming compromised), or erasing and reinstating them.
If any build agents have indeed been compromised, any illicit actions would have been performed through the user account configured for the TeamCity agent process. This user account may have privileged access to other resources on the build machine, or in the case of a domain user account, additional resources on your local network.
Unknown running processes
While you carry out further checks, consider stopping all unknown background processes on the host operating system.
Audit logs
The Audit screen in TeamCity provides a wealth of information regarding all configuration changes and security actions. Check the entries on the Audit screen for the time period you believe the server was compromised.
You can filter the list of audit items to a particular subsystem, such as User actions. This is discussed further in the following sections.
Log files can also be accessed directly on the file system under the logs folder of the TeamCity server home directory.
If we take the case of CVE-2024-27198, this particular vulnerability enabled attackers to use a specially formed HTTP request to bypass authentication checks and enable the unauthenticated use of the REST API. These specially formed requests included ;.jsp in the URL. The regular expression ;\S*\.jsp\?\S*jsp= can help check for potential matches of these types of requests within the various teamcity-javaLogging log files.
It is also important to note that attackers may attempt to cover their tracks by wiping or modifying log files, which means that the absence of such requests in the log files does not necessarily mean the server wasn’t compromised using this approach.
Service credentials
TeamCity often has privileged access to external services. The server holds connection settings for those services, which could include any of the following:
- Version control systems – Git, GitHub, Perforce, etc.
- Notifications – Slack, SMTP server, etc.
- Cloud profiles – for spinning up builds on AWS, Azure, Google Compute Engine, etc.
- Registries – NPM, Docker, etc.
- Issue trackers – Jira, GitHub Issues, YouTrack, etc.
- Other connections – AWS for issuing temporary credentials to agents.
There may also be service credentials stored as custom configuration parameters in projects and build configurations.
All of these services should have their credentials rotated, and an audit should be performed on them for any unrecognized actions or changes, especially on repositories within your version control system.
Users and credentials
An attacker may have created one or more user accounts. Check the Audit screen, filtering for User actions | Create/Delete/Update.
If the credentials of any users have been changed (according to the audit log), then those users should have their credentials rotated. Any newly created access tokens should also be removed (see the next section).
Tokens
A common vector point in recent attacks has been the unauthorized creation of authentication tokens. We recommend you remove all authentication tokens associated with users in the system.
It’s also important to consider that an attacker may have created a token but then subsequently deleted it. You can analyze the history of created and deleted tokens by checking the Audit screen and filtering for User actions.
Authentication modules
Check the authentication settings to see if any of the settings have been changed.
Verify any single sign-on authentication modules to see if their settings have been modified. For example, if you’re using the Google authentication module, check whether any additional allowed domains have been added to its settings.
Projects and build configurations
An attacker may have created new projects or build configurations. Check all projects and build configurations to ensure that they are recognized and haven’t been modified.
Check the settings for changes to build features such as File Content Replacer, which may have been modified by an attacker to alter the contents of files used during the build process.
These types of configuration changes can also be monitored via the audit logs, or in the repository if versioned settings are enabled for the project.
Build artifacts
If build agents have been tampered with (see the section above), or the project and/or its build configurations have been modified, any build artifacts generated from the time the initial compromise took place through to the present time should be considered unsafe.
You should consider all types of artifacts. For example, if you are using the Build Cache feature, there’s a risk that any recently generated caches could be compromised.
Internal properties
Check for any unknown internal properties under Administration | Diagnostics, or in the internal.properties file in the data directory. Internal properties are often experimental switches suggested by our Support team or toggles for fine-tuning very specific features, so the presence of unexpected internal properties is an indication that an attacker might have successfully compromised the server.
One indicator of compromise might be the presence of the following internal property enabling the execution of processes on the host server through the REST API for diagnostic purposes:
rest.debug.processes.enable=true
The above property has no effect on versions of TeamCity starting from 2023.11.3, but may still have been applied by an attacker. If you are on a version of TeamCity older than 2023.11.3, the teamcity-activities.log on the server may contain information regarding any scripts and commands executed by attackers.
Remove any unknown internal properties, or reach out to our Support team if you aren’t certain about any of them. By default, no internal properties are added in TeamCity.
Plugins
TeamCity supports a plugin ecosystem, where additional plugins can be added to the server. If a plugin has build agent capabilities, it may also install dependencies on connected build agents.
Check for any unknown plugins. This can be verified in the user interface by navigating to the Plugins screen and checking the plugins list. You can also check the /plugins folder within the server’s data directory and note the modified/created dates of any suspicious plugins.
It’s important to consider that an attacker may have uploaded a plugin but then subsequently deleted it, leaving no trace on the file system. However, you can also check the Audit screen for any entries detailing the creation or deletion of plugins. You can do this by filtering for Server actions | Plugins on the Audit screen.
Tools
TeamCity enables you to define tools that can be automatically installed on build agents. Check for unknown tools by going to the Tools screen.
You can also check for any tools by looking directly in the server’s data directory, under /plugins/.tools.
Check whether any tools were added and then subsequently removed by an attacker (in an attempt to cover their tracks). Use the Audit screen and filter for Tool actions.
Bringing the environment back online
If you’re unsure whether the TeamCity server still contains any remnants of an attack, or if it becomes evident that it’s unsuitable for continued use, you may wish to consider creating a fresh TeamCity environment and only copying across the necessary configuration files. Our guide on creating a copy of a TeamCity server along with all its data can help you with this process.
If you do copy TeamCity data from the compromised server to a new environment, ensure you have performed a thorough check of all files being transferred.
Always ensure that you have installed the latest version of TeamCity, or that you’ve applied any security patches, prior to bringing the TeamCity environment back online.
Ensure the TeamCity server process is operating with only the minimum necessary permissions. For Windows setups, avoid using an administrator account (including SYSTEM) to run the service. In other environments, refrain from running the server process under an account with root access.
Internet-facing TeamCity servers
Unless you can ensure the prompt application of security patches or updates as soon as they are released, it is much safer to isolate your TeamCity server and its build environment from the internet.
If your TeamCity server is internet-facing (i.e. accessible to external users who can reach the TeamCity server login screen), consider only allowing access to users through a VPN connection or implementing an additional layer of security to prevent the server from being reached by unauthorized users. Even exposing the TeamCity login screen and the REST API can provide potential entry points for attackers to exploit vulnerabilities as soon as they are discovered.
Support
The list of recommendations above isn’t exhaustive. If you believe your TeamCity On-Premises environment is still compromised in some way, we recommend consulting a security professional.
You can also contact the TeamCity Support team if you have any further questions or concerns after following the guidance above.
Subscribe to TeamCity Blog updates
