Post-Mortem of YouTrack Hosted Availability Issue

Valerie Andrianova

As you already know, YouTrack is hosted at Amazon Elastic Cloud (Amazon Elastic Compute Cloud (N. Virginia)). AWS has been experiencing huge connection problems since Thursday, April 21, 2011 which meant that all YouTrack hosted instances were unavailable.
We would like to provide you with details of the issue we’ve experienced and decisions we had to make to solve the problem fast and effective.
AWS connection problems started on Thursday, April 21, at 10.00 GMT+1. Since it was the first time we were experiencing such a long service unavailability while using Amazon services, we decided to wait till the problem would be solved by Amazon. We’ve been waiting for about 24 hours, and still could not connect to Amazon EBS volumes.
On Friday, April 22, at 9.00 GMT+1, we decided to grab the latest backup we could possibly get from Amazon. It was a backup taken on Thursday, April 21, at 7.00 GMT+1, 3 hours before the connection was broken. Also, we decided to move our servers to another Amazon region (California), which was up and running stable at the moment. It took us around a day to move all servers and start all instances. So, on Friday, April 22, at around 21.00 GMT+1, all YouTrack hosted servers were up and running and all instances were available. According to our statistics, all servers are running normally since Friday night.
Please, contact our technical support if you still have any problems with your YouTrack instance.
Also, please let us know if you would like us to restore your data lost between the backup made and Amazon unavailability (Thursday, April 21, 7.00-10.00 GMT+1) time.
Conclusions: Once again, we apologize for the inconvenience. We’ve done everything possible to solve the problem fast and effective. As this is the fist time Amazon had that huge connection problems, definitely, it’s a good lesson to learn and just the right time to make conclusions. So, based on this experience, we decided to create technology that would prevent us from loosing any data between the backups made. We expect to implement it before the official release of YouTrack hosted.

Sincerely,
JetBrains YouTrack Team

Subscribe to YouTrack Blog updates

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!