Dmitry Jemerov's Weblog

Exception processing and Exception Analyzer

After the release of IntelliJ IDEA 5.0, from time to time we receive feedback telling us that IntelliJ IDEA 5.0 is less stable than the previous versions, that the exception reports submitted by users are now ignored, that we’re not paying adequate attention to the product quality, and so on. As quite a lot of time of the team during this and past week was spent on fixing bugs for the IntelliJ IDEA 5.1 release, I think that now should be a good time to explain the system we have recently set up for processing exception reports.

First of all, a very important thing to note is that in all versions of IntelliJ IDEA before 5.0 the exception reporter was disabled in release builds. Because of that, in previous versions the users simply didn’t see the crashes – some operations failed, or some analysis produced incorrect results, silently. In the new version, we’ve become sufficiently confident of the product quality to enable the exception reporter in release builds, not afraid of being overwhelmed by the number of reports.

To help us deal with the exception reports we’ve been getting, we decided to create an internal tool called Exception Analyzer. The current version of Exception Analyzer was mostly developed by me, so that’s one more chance for me to tell about my own work. :-)

The main concepts of the Exception Analyzer are “problem” and “report”. A problem is, essentialy, a bug in IDEA or in some other code (for example, JDK) which causes exceptions to appear, and a report is an occurrence of a problem at a user’s machine which has been submitted to us. Our old tracker didn’t have this distinction: every report became a separate bug, we often got different instances of the same problem assigned to different developers, and marking them as duplicates required a lot of manual work.

The Exception Analyzer consists of two main components: a server-side component which keeps a database of problems and reports and processes newly submitted reports, and a plugin for IntelliJ IDEA which can be used to associate new reports with problems and to investigate the reported problems. The components communicate via the SOAP protocol.

The old IDEA ITN tracker is currently used solely as the receiver and storage of problem reports. This allows us to keep the existing submit system and to process reports from versions of IDEA released before the Exception Analyzer was created. Exceptions in ITN are currently never resolved or otherwise processed; the real activity happens elsewhere.

When a new report is submitted to ITN, the Exception Analyzer first unscrambles the stacktrace, based on the scrambling map of the build in which the problem occurred. Then it checks if it matches the signature of an already known problem. If it does, the report is automatically linked to the problem, and the user receives a comment with a link to the JIRA issue describing the problem.

If the report is not recognized, it is shown in the “New Reports” view of the IDEA plugin, waiting to be reviewed by a developer. Developers regularly review the new reports, create new problems if the reports are indeed something new, and assign the problems to developers. As soon as a problem is created, a JIRA request is created to track the problem status and resolution, and the link to the request is sent to the user. Also, a signature (part of stacktrace) can be specified for a problem, so that other reports containing the same stacktrace fragment are linked to the same problem. Other new reports matching the signature can be automatically linked to the problem. The JIRA activity related to exceptions is tracked in the jetbrains.intellij.jira.idea.exceptions newsgroup.

Currently the Exception Analyzer is created as an internal tool, hard-coded to work with our database and project structure, but we have considered including this functionality in the team server in some way or another. The decision on this topic depends on how much interest there is in this functionality, so if you would like to use a similar tool for your own project, feel free to contact us and tell us about your requirements.

Posted by Dmitry Jemerov at December 23, 2005 08:07 PM