YouTrack InCloud Upgrade Issue Post-Mortem
Last Monday, August 24, we upgraded the first part of our InCloud servers to YouTrack 6.5. On Tuesday, August 26, our clients started to report some problems connected with the upgrade.
The problems included:
1. YouTrack failed to start
2. YouTrack failed to renew the license
3. Integration with Hub was broken which led to issues with user management.
As you already know, starting from version 6.0 YouTrack ships with Hub. For YouTrack InCloud, this means we have a set of YouTrack instances running on one server and a set of corresponding Hub instances running on another server.
For the version 6.5 upgrade we applied a different clustering algorithm to our servers. When we upgraded several YouTrack servers to 6.5 and the corresponding Hub servers to 1.0, it appeared that some Hub 1.0 instances on the upgraded servers were connected with YouTrack instances that weren’t yet upgraded. YouTrack 6.0 could not correctly work with Hub 1.0, plus there was an issue with renewing YouTrack licenses in Hub 1.0.
1. We have upgraded all the impacted YouTrack servers to the version 6.5. As we started this upgrade from the US servers, we’ve had to move almost all the US InCloud customers to 6.5.
2. We have fixed the issue with automatic license renewal. However, if your license has expired and you haven’t renewed it in time, your YouTrack instance might be completely unavailable until you renew your subscription.
3. We have manually fixed the problems with Hub and YouTrack desynchronization for those who have contacted us. If you detect any issue with user and permission management on your instance after the upgrade, please contact our support team.
Measures to prevent in future
YouTrack and the corresponding Hub instances are now tightly connected. The next time we will upgrade them in pairs only. The license renewal mechanism has now been doubly tested from both YouTrack and Hub sides.
Please accept our deepest apologies for any inconvenience experienced due to these upgrade issues. Do not hesitate to contact us in case you still have any problems.