As you might have noticed, on Wednesday, November 16, we experienced the problem with our own JetBrains YouTrack installation. The problem caused significant slowdown of the tracker and its further unavailability for users.
What went wrong
In our tracker we have a number of custom workflows, attached to different projects. Due to erroneous configuration, one of the workflow rules got into infinite loop while generating notifications, and started to increase loading of the server, which led to system’s slowdown.
Fixing the problem
Being unable to stop executing the rule via UI, due to its unavailability, we had to restart the server, which caused system downtime.
Lessons learned and preventative actions
In order to prevent this problem in future, we decided to add the timeout for the workflow rules execution. This issue will be implemented ASAP.
We apologize for the inconvenience you might have experienced due to our system downtime. Though, it was a very good lesson to learn from, and we’ll do our best to prevent our customers from experiencing this kind of issues with their own YouTrack installations.
JetBrains YouTrack Team
TagsAgile Agile board baking boards beta Bitbucket bug fix commands conference Daring demo EAP GitHub Integration GitLab How we Scrum Import InCloud integration JetBrains Jira import Kanban makeitworkflow post-mortem Promotion RC release REST API roadmap Scrum search TeamCity time tracking update webinar Workflow workflow editor YouTrack YouTrack 4.1 YouTrack 4.2 YouTrack 5.0 YouTrack 6.0 YouTrack 6.5 YouTrack 6.5 EAP YouTrack 7.0 YouTrack Daring YouTrack InCloud
Other JetBrains Blogs