TeamCity
Powerful CI/CD for DevOps-centric teams
Preventing Exploits: JetBrains’ Ethical Approach to Vulnerability Disclosure
Introduction
This blog post follows up on the recent TeamCity On-Premises security vulnerabilities notification and the subsequent post that described our timeline for addressing those vulnerabilities.
At JetBrains, we adhere to a carefully balanced approach to vulnerability disclosure. We follow our Coordinated Disclosure Policy that prioritizes the safety of our customers and the integrity of their data. We believe in the importance of transparency, yet we are acutely aware that releasing full technical details of vulnerabilities (especially how to exploit them) simultaneously with a fix can lead to more harm than good.
This blog post outlines the potential consequences of such disclosures and explains our methodology for sharing just enough information to inform our customers, without unnecessarily broadening the window of opportunity for attackers. By doing so, we aim to strike a delicate balance between prompt disclosure and minimizing the risk of widespread exploitation when a fix is released.
Background
On February 20, 2024, two vulnerabilities in TeamCity On-Premises were disclosed to JetBrains by Rapid7. Rapid7 is a cybersecurity company that provides a range of security solutions, including vulnerability management, application security, and incident detection and response. There was no prior paid engagement between JetBrains and Rapid7 that led to the discovery of these vulnerabilities, so we are thankful for Rapid7’s help in discovering these vulnerabilities and reporting them to us in a responsible manner.
As a CVE Numbering Authority (CNA), we assigned the CVE identifiers CVE-2024-27198 and CVE-2024-27199 within one day of receiving the report, as we always intend to make the issues public from when they’re first reported. CNAs can’t assign CVE IDs without making CVE records public, according to CNA rules. Issue CVE-2024-27198 was the more critical of the two, as it allowed for a complete authentication bypass. This meant an attacker could perform administrative actions on the server and its host environment.
We communicated with Rapid7 and discussed a release strategy. Rapid7 insisted on following their disclosure policy, which meant a full disclosure of the vulnerabilities and how to exploit them would be made alongside the release of a fixed TeamCity version. Ultimately, JetBrains made the decision not to make a coordinated disclosure with Rapid7.
We firmly believe that releasing the full technical details of a vulnerability and the exploit steps simultaneously with its fix is entirely unethical and harmful to our customers, provided that enough details are made publicly available to allow customers to fully understand the risks and protect themselves against the vulnerability.
Releasing a full disclosure and providing the exploit steps enables potential attackers to immediately exploit a vulnerability before any customers have had the opportunity to patch their environments.
The release
Both vulnerabilities were fixed by the release of TeamCity 2023.11.4 on March 4, 2024, along with the release of a security patch for customers who couldn’t upgrade to the latest version or who wanted to quickly patch their environments.
We immediately notified all TeamCity On-Premises customers and JetBrains Security Bulletin subscribers to warn them of the security vulnerabilities and the availability of a fixed version and a security patch. We also published a blog post describing the vulnerabilities and detailing the CVE numbers. In these advisories, we strongly encouraged customers to upgrade or install the security patch urgently. In addition, the vulnerability details were added to our Fixed security issues page, and both CVE records were made public.
Rapid7 noted our release, which triggered them to publish a full disclosure later on the same day. This small delay provided many of our customers a small window of opportunity to upgrade or patch their environments prior to the full details of the exploits being published by Rapid7.
We are aware of many customers who were able to apply the security patch or upgrade prior to the exploits being published by Rapid7.
Unfortunately, many others were not so fortunate.
The aftermath
Rapid7 made their full disclosure, which included detailed exploit steps, only five hours after we made our fixes available to customers, followed by publishing an exploit module shortly after. After the full disclosure was made, we started hearing from some customers who were noticing that their servers had been compromised.
This was due to the immediate availability of publicly documented exploit examples published by Rapid7, which meant attackers of any skill level had all the resources they needed to quickly exploit the vulnerabilities in the wild.
Here is what some of our customers have reported after the full disclosure and availability of public exploits by Rapid7:
Customer A
- Believed they were impacted by the CVE-2024-27198 vulnerability.
- Files on their TeamCity server were all encrypted and a ransomware note was left on the machine.
Customer B
- They try to keep their server updated whenever security vulnerabilities are reported to them. However, they weren’t able to react fast enough to this particular incident.
- They noticed several unauthorized admin accounts created on the server.
Customer C
- Their TeamCity environment had been compromised through the recent vulnerabilities.
- By the time they were able to lock down the server, the attackers had already gained access and had started encrypting files as part of a ransomware attack.
- Luckily, they caught the attack relatively early as the TeamCity database was intact and most of the configuration files were unaffected.
- Unfortunately, their build artifacts had all been encrypted from the ransomware attack.
Customer D
- They woke up on March 5 to notifications of the security vulnerabilities in TeamCity.
- Several unknown user accounts had been created on their TeamCity server.
- They took the server offline as soon as they realized what was happening, and upgraded it.
- From their analysis of log files, they came to the conclusion that the attackers were intending to use the server in DDoS attacks, rather than targeting the contents of the server itself.
Considerations
Based on the examples above, we can outline some important considerations regarding the different approaches to responding to security incidents.
Ethics of disclosure
When full technical details of vulnerabilities and exploit examples are disclosed at the same time as fixes, this raises important questions about the balance between transparency and security, as well as about the impact these actions have on the wider community.
It is important to consider the implications for all parties involved, including how any actions can affect the overall security of organizations who are potentially at risk.
We fully support the timely disclosure of vulnerability details when a fix is released. However, we provide only the necessary details for customers to understand the vulnerability’s scope and severity, enabling them to take the appropriate actions, but without disclosing so much information that it facilitates straightforward exploitation.
We are also fully committed to disclosing complete vulnerability details (and the exploit steps) to benefit security researchers, but only after we see a significant number of customers have actually upgraded their systems to safe versions or installed patches.
Early release of exploits
Other vulnerabilities in JetBrains products without corresponding public exploits have not been exploited nearly as commonly or quickly as CVE-2024-27198 and CVE-2024-27199 have been.
The immediate availability of exploit examples does not consider the varied time zones of affected organizations, their resources, and the time required for them to successfully apply any patches or updates.
Reverse engineering of patches
Attackers can, and do, reverse-engineer patches. However, this is a complex process that requires both time and high technical skill. Skilled security professionals or attackers must delve deep into the code, understanding both its intended functionality and the entry points that could be exploited.
The availability of exploit details essentially circumvents the need for reverse engineering. When full exploit steps are made publicly available, it removes the requirement for attackers to possess in-depth knowledge or skills. This significantly lowers the barrier to entry, allowing a much wider audience, including less-skilled attackers, to exploit the vulnerabilities. The immediate availability of such information can lead to widespread attacks before the majority of organizations have had the chance to apply any patches (as seen in this instance), dramatically increasing the risk posed by the vulnerability.
By obfuscating patches and refraining from providing detailed exploit steps, we make the process of discovering and exploiting vulnerabilities significantly more challenging and time-consuming for potential attackers. This approach buys time for organizations to apply patches before they are widely exploited. Although vulnerabilities will eventually be understood and possibly exploited, this strategy aims to manage the timeline in a way that favors defense over attack.
With CVE-2024-27198 and CVE-2024-27199, we took steps to make the patch analysis harder for attackers through techniques such as obfuscation, which would have usually allowed more time for many customers to patch their environment. However, the full disclosure by Rapid7 removed that time gap without adding any value to our customers and users.
Our approach
When a vulnerability is discovered in a JetBrains tool, we will always follow our Coordinated Disclosure Policy.
Once a fixed version becomes available, we notify our customers about the vulnerability and recommend they update to a more secure version of the product. This notification provides actionable information through which a customer can understand the high-level details of the vulnerability and its risk, such as the CVE information and a summary of the vulnerability’s attack vector (the entry point).
This approach ensures our customers are informed of the risk and are able to take action with a much lower risk of being exploited.
Once customers have been provided with a reasonable time period to patch or upgrade, we are then open to sharing a full disclosure of vulnerabilities and the exploit details.
We firmly believe this provides an ethical approach to vulnerability disclosure by providing customers with a window of opportunity to patch or upgrade their systems before the full technical details of vulnerabilities and their exploits are made publicly available.
Common industry practices
The type of policy we follow is commonplace throughout the industry.
For example, the Project Zero team at Google follows a 90+30 day policy that states:
“A vendor has 90 days after Project Zero notifies them about a security vulnerability to make a patch available to users. If they make a patch available within 90 days, Project Zero will publicly disclose details of the vulnerability 30 days after the patch has been made available to users.”
Microsoft’s Coordinated Vulnerability Disclosure policy states (emphasis in bold is ours):
“Some companies and individuals believe that, even if there is no evidence of an attack using a found vulnerability, earlier public disclosure is preferred because it forces vendors to develop mitigations more quickly. However, given the facts that vendors are in the best position to assess the risk priority of different vulnerabilities and that most users are reliant on a software provider to release a patch, Microsoft believes that those who disclose a vulnerability before a fix is broadly available are doing a disservice to millions of people and the systems they depend upon. Furthermore, even for those who take action, risk is significantly increased when information that a criminal can use to orchestrate an attack is publicly released. Our research shows that of the vulnerabilities privately disclosed and fixed through coordinated disclosure practices each year by all software vendors, almost none are exploited before a “fix” has been provided to customers, and even after a “fix” is made publicly available, only a small number are ever exploited.”
The Open Worldwide Application Security Project (OWASP) has a Vulnerability Disclosure Cheat Sheet that states:
“Some organizations may request that you do not publish the details at all, or that you delay publication to allow more time to their users to install security patches. In the interest of maintaining a positive relationship with the organization, it is worth trying to find a compromise position on this.”
They go on to say:
“For more serious vulnerabilities, it may be sensible to ask the researcher to delay publishing the full details for a period of time (such as a week), in order to give system administrators more time to install the patches before exploit code is available.”
Conclusion
In summary, JetBrains’ commitment to vulnerability reporting and disclosure is underscored by our Coordinated Disclosure Policy, which balances the need for transparency with the requirement of protecting our customers. By carefully timing the disclosure of vulnerabilities and associated exploits, we provide a critical window for mitigation, aligning with best practices in the industry. This approach both helps in safeguarding our customers from potential threats and fosters an environment of trust and security within the community.
We recognize the importance of collaboration and ethical responsibility in addressing vulnerabilities. Our policy is designed to give customers ample time to respond to vulnerabilities, thereby reducing the window of opportunity for attackers.
JetBrains remains dedicated to continually improving our products and security practices, with our customers’ safety being our top priority. We believe that a responsible and coordinated approach to vulnerability disclosure is fundamental to building a safer digital world for everyone. Our commitment to this principle is unwavering, and we will continue to adapt and refine our approaches to address the ever-evolving nature of security vulnerabilities.
Subscribe to TeamCity Blog updates
