Post-Mortem of JetBrains YouTrack Zombie-Mode Issue

As you might have noticed, on Wednesday, November 16, we experienced the problem with our own JetBrains YouTrack installation. The problem caused significant slowdown of the tracker and its further unavailability for users.
What went wrong
In our tracker we have a number of custom workflows, attached to different projects. Due to erroneous configuration, one of the workflow rules got into infinite loop while generating notifications, and started to increase loading of the server, which led to system’s slowdown.
Fixing the problem
Being unable to stop executing the rule via UI, due to its unavailability, we had to restart the server, which caused system downtime.
Lessons learned and preventative actions
In order to prevent this problem in future, we decided to add the timeout for the workflow rules execution. This issue will be implemented ASAP.

We apologize for the inconvenience you might have experienced due to our system downtime. Though, it was a very good lesson to learn from, and we’ll do our best to prevent our customers from experiencing this kind of issues with their own YouTrack installations.

Kind Regards,
JetBrains YouTrack Team

image description