Post-Mortem of JetBrains YouTrack Zombie-Mode Issue

Valerie Andrianova

As you might have noticed, on Wednesday, November 16, we experienced the problem with our own JetBrains YouTrack installation. The problem caused significant slowdown of the tracker and its further unavailability for users.
What went wrong
In our tracker we have a number of custom workflows, attached to different projects. Due to erroneous configuration, one of the workflow rules got into infinite loop while generating notifications, and started to increase loading of the server, which led to system’s slowdown.
Fixing the problem
Being unable to stop executing the rule via UI, due to its unavailability, we had to restart the server, which caused system downtime.
Lessons learned and preventative actions
In order to prevent this problem in future, we decided to add the timeout for the workflow rules execution. This issue will be implemented ASAP.

We apologize for the inconvenience you might have experienced due to our system downtime. Though, it was a very good lesson to learn from, and we’ll do our best to prevent our customers from experiencing this kind of issues with their own YouTrack installations.

Kind Regards,
JetBrains YouTrack Team

Subscribe to YouTrack Blog updates

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!