post-mortem

Last Monday, August 24, we upgraded the first part of our InCloud servers to YouTrack 6.5. On Tuesday, August 26, our clients started to report some problems connected with the upgrade. The problems included: 1. YouTrack failed to start 2. YouTrack failed to renew the license 3. Integration with Hub was broken which led to issues with user management. What Happened? As you already know, starting from version 6.0 YouTrack ships with Hub. For YouTrack InCloud, this means we have a set of YouTrack instances running on one server and a set of corresponding Hub instances running o

Last Friday, August 7, around 5 PM GMT we experienced a security issue on our production instance at youtrack.jetbrains.com. For about an hour, all external users and guest accounts obtained permissions to all of our projects, including internal ones. Issues, comments, and attached files in public projects that were restricted by permitted user group were not affected. The issue was caused by an administration failure during the permission synchronization process. The issue was detected and fixed within an hour thanks to our Operation team. We've also received bug reports from our extern

On Monday September, 23rd YouTrack InCloud servers were updated to v5.0.3. As some of you might noticed, this update caused an issue related to custom domains. Luckily, it affected only several custom domains. Due to the issue custom domains were reset to default "...".myjetbrains.com address. YouTrack engineers investigated the problem and came up with a quick solution how to set custom domains back. However, in order to apply the bug fix we had to restart the servers. Therefore, some of you might noticed another unscheduled 15 minutes downtime. We apologize for the inconvenience you mi

As you might have noticed, on Wednesday, February 20, there was the problem with YouTrack InCloud availability. The problem caused downtime at around 19.00 GMT and another downtime at around 23.00 GMT. What was the problem We use Nginx server which is focused on high concurrency, performance and low memory. On February 20 Nginx was unusually high loaded. This caused Out of memory error and we decided to restart Nginx on another server. We got 30 minutes downtime, finally. Honestly, longer than we expected. What we've already done to solve this issue Our Nginx was restarted on anoth

As you might have noticed, on Wednesday, November 16, we experienced the problem with our own JetBrains YouTrack installation. The problem caused significant slowdown of the tracker and its further unavailability for users. What went wrong In our tracker we have a number of custom workflows, attached to different projects. Due to erroneous configuration, one of the workflow rules got into infinite loop while generating notifications, and started to increase loading of the server, which led to system's slowdown. Fixing the problem Being unable to stop executing the rule via UI, due to its u

As you already know, YouTrack is hosted at Amazon Elastic Cloud (Amazon Elastic Compute Cloud (N. Virginia)). AWS has been experiencing huge connection problems since Thursday, April 21, 2011 which meant that all YouTrack hosted instances were unavailable. We would like to provide you with details of the issue we've experienced and decisions we had to make to solve the problem fast and effective. AWS connection problems started on Thursday, April 21, at 10.00 GMT+1. Since it was the first time we were experiencing such a long service unavailability while using Amazon services, we decided to w