December 23, 2005

Exception processing and Exception Analyzer

After the release of IntelliJ IDEA 5.0, from time to time we receive feedback telling us that IntelliJ IDEA 5.0 is less stable than the previous versions, that the exception reports submitted by users are now ignored, that we’re not paying adequate attention to the product quality, and so on. As quite a lot of time of the team during this and past week was spent on fixing bugs for the IntelliJ IDEA 5.1 release, I think that now should be a good time to explain the system we have recently set up for processing exception reports.

First of all, a very important thing to note is that in all versions of IntelliJ IDEA before 5.0 the exception reporter was disabled in release builds. Because of that, in previous versions the users simply didn’t see the crashes – some operations failed, or some analysis produced incorrect results, silently. In the new version, we’ve become sufficiently confident of the product quality to enable the exception reporter in release builds, not afraid of being overwhelmed by the number of reports.

To help us deal with the exception reports we’ve been getting, we decided to create an internal tool called Exception Analyzer. The current version of Exception Analyzer was mostly developed by me, so that’s one more chance for me to tell about my own work. :-)

The main concepts of the Exception Analyzer are “problem” and “report”. A problem is, essentialy, a bug in IDEA or in some other code (for example, JDK) which causes exceptions to appear, and a report is an occurrence of a problem at a user’s machine which has been submitted to us. Our old tracker didn’t have this distinction: every report became a separate bug, we often got different instances of the same problem assigned to different developers, and marking them as duplicates required a lot of manual work.

The Exception Analyzer consists of two main components: a server-side component which keeps a database of problems and reports and processes newly submitted reports, and a plugin for IntelliJ IDEA which can be used to associate new reports with problems and to investigate the reported problems. The components communicate via the SOAP protocol.

The old IDEA ITN tracker is currently used solely as the receiver and storage of problem reports. This allows us to keep the existing submit system and to process reports from versions of IDEA released before the Exception Analyzer was created. Exceptions in ITN are currently never resolved or otherwise processed; the real activity happens elsewhere.

When a new report is submitted to ITN, the Exception Analyzer first unscrambles the stacktrace, based on the scrambling map of the build in which the problem occurred. Then it checks if it matches the signature of an already known problem. If it does, the report is automatically linked to the problem, and the user receives a comment with a link to the JIRA issue describing the problem.

If the report is not recognized, it is shown in the “New Reports” view of the IDEA plugin, waiting to be reviewed by a developer. Developers regularly review the new reports, create new problems if the reports are indeed something new, and assign the problems to developers. As soon as a problem is created, a JIRA request is created to track the problem status and resolution, and the link to the request is sent to the user. Also, a signature (part of stacktrace) can be specified for a problem, so that other reports containing the same stacktrace fragment are linked to the same problem. Other new reports matching the signature can be automatically linked to the problem. The JIRA activity related to exceptions is tracked in the jetbrains.intellij.jira.idea.exceptions newsgroup.

Currently the Exception Analyzer is created as an internal tool, hard-coded to work with our database and project structure, but we have considered including this functionality in the team server in some way or another. The decision on this topic depends on how much interest there is in this functionality, so if you would like to use a similar tool for your own project, feel free to contact us and tell us about your requirements.

Posted by Dmitry Jemerov at December 23, 2005 08:07 PM

This is a test post to check if the comment blacklist system is working properly.

Posted by: Dmitry Jemerov at December 26, 2005 03:25 PM

This "Exception Analyzer" sounds very good and IMHO it would be a good idea to "publish" it to the public.
Different teams use different bug tracking tools(and tool combinations), so I suppose you can't do it for all of them, so IMHO you have the following choices:
1. Make the tool generic, with "complex" configuration to allow to "map" it's usage for many bug trackers (this sounds very complicated).
2. Publish the *entire* tool as a plug-in with sources(and documentation) and call it e.g. "JIRA Exception Analyzer". This will allow others to write similar plug-ins for other tracking tools.
3. A combination of 1 and 2: Make the tool a little more generic, and publish it with sources. This would allow users to start by mostly extending it, and not from 0 like in the case of 2 where the JIRA plug-in would be just an example.

I for one would preffer variant 3 cause many companies have their own "in house" tracking tool developed over the years, so the IntelliJ Exception Analyzer would be a great improvement.

IMHO such a tool would be very important cause the work effort with trackers increases exponentially with the number of issues, and such a tool would help to cut down the work effort.

Thanks in advance,


Posted by: Ahmed Mohombe at December 26, 2005 03:43 PM

I wonder what happens with the messages typed in by the user reporting the exception. Sometimes a particular exception is extremely easy to reproduce for example and that information could be valuable. Do these messages find their way to the developer responsible for the JIRA request?

Posted by: Bas Leijdekkers at December 29, 2005 12:36 PM

I like this. I think the "community source" project that I work on should do the same thing. Was this included in TeamCity?

It seems like this kind of thing should be available somewhere, it's so useful.

Can it accept reports on plugins too, if the plugin developer wants to review such reports? I guess that developer would be taking up resources on your JIRA server, then, but it would help improve the quality of IDEA's plugins.

Posted by: J. David Beutel at October 20, 2006 01:48 AM
Post a comment

Remember personal info?