Writing a Java Inspection for IntelliJ IDEA Using AI Assistant

Tagir Valeev

We’ve just released AI Assistant for IntelliJ IDEA, which can help you complete many tasks in the IDE, such as explaining code, generating commits, and assisting with writing code. The AI Assistant plugin is powered by the JetBrains AI Service, which uses different large language models (LLMs) under the hood and enables specific AI-powered features within many JetBrains products.

Since we like dogfooding, I decided to try out AI Assistant in IntelliJ IDEA to write a new Java inspection for IntelliJ IDEA, and write down my experience while solving this task. Let’s take a look!

What do we want to create with AI Assistant?

Yesterday, a coworker approached me and asked about making a simple new Java inspection:

Hello! Am I missing something, or is there no inspection in IDEA like this?
(int)($VAR$ ^ $VAR$ <<< 32) => Long.hashCode($VAR$)

In the Java team, we usually give a task like this to new team members because it’s a relatively easy, isolated, and well-formulated problem and it helps them to learn about the IntelliJ IDEA codebase. Now, everybody says that AI can replace programmers, 80% of new code is written by AI, and so on. So I thought: why not ask our AI Assistant to do this for me? It was 16:09.

My response was:

It looks easy enough to ask a neural network…
I wonder if it can do it.

Asking AI Assistant to write code for a new Java inspection

In the chat window, I ask AI Assistant to perform this task, just copying the desired replacement from the original request. It produces some code for this new inspection:

As requested in the initial prompt, AI Assistant also suggests a corresponding quick-fix:

As you can see, some of the generated methods are incomplete, and their implementation is missing. This is not a problem, as you can ask AI Assistant to write these as well:

Honestly, every time I ask AI Assistant to generate the next method, I expect the answer to be something like, “Come on, should I do the whole job instead of you? Write some code yourself!” Luckily, it is always ready to help:

Once all the methods are generated, I have what I need to paste into the editor, register the extension, and compile and launch it, all without writing any code manually. First, I register the extension myself, because this is mostly a copy-paste of an existing extension with minor edits. AI Assistant would not be faster for me here.

Can AI Assistant write tests? A typical test case for inspections in IntelliJ IDEA requires creating a test class (which is also mostly a copy-paste of an existing test) and then creating beforeXyz.java and afterXyz.java to ensure that the quick-fix works correctly. Since the test data can be relatively straightforward, let’s ask AI Assistant to do this!

An inspection also needs textual descriptions that can be shown in documentation – another thing the LLM should do well.

While AI Assistant is able to write some HTML for us, the writing style it generates differs from what we have for other inspections. In fact, we have a template for these descriptions, so why not feed the template into a prompt?

This looks much better!

All of the resulting code actually compiles and does something, though the test doesn’t pass. What’s the problem?

The first and most important thing is that the problem was reported incorrectly, and thus the initial prompt was incorrect as well. There’s no <<< operator in Java! It should be >>> instead; the unsigned right shift operator. I would expect a real programmer who’s given this task to detect the typo and fix it before starting to solve the problem.

Unfortunately, the LLM started solving it right away, without first analyzing whether the prompt was correct. And it actually tries to match << using a constant JavaTokenType.LTLT, which we use for this token inside IntelliJ IDEA sources (“LT” means “less than” and is used to designate “<”). There’s apparently no LTLTLT constant, as there’s no <<< Java token, so AI Assistant tried to fix the mistake. However, it would be correct to use JavaTokenType.GTGTGT, which corresponds to >>> (“GT” means “greater than” and is used to designate “>”), so I fixed this. Additionally, I need to replace all instances of <<< with >>> in the tests and the description.

Once that’s done, it is better. Can we just commit and push the result? Unfortunately, the quality of the solution is in fact not very satisfactory, and there’s room for a number of improvements:

1. The inspection should still work if the original code has extra parentheses, like (int)(($VAR$) ^ ($VAR$ >>> 32)). To support this, we should add some calls to skipParenthesesExprDown.
2. The inspection matches only when the $VAR$ part is a reference, which looks like an unnecessary limitation. It could be for example, an array element, and we should also match that case.
3. It compares two instances of $VAR$ textually, but it would be more robust to compare on the abstract syntax tree level. We have a utility for this called EquivalenceChecker, which has existed since at least 2005.
4. It doesn’t match the equivalent (var >>> 32) ^ var (XOR arguments are swapped). We would like to match this as well.
5. The static method Long.hashCode was introduced in Java 8. So if we suggest it when the user’s project is configured to use Java 7 or older, there will be a compilation error. The inspection should not suggest anything on code older than Java 8.
6. Messages are not externalized, which means localizing them is not possible. This is probably OK, as it’s likely hard for the LLM to generate changes in several files. And we have a quick-fix for this, so we don’t actually need to write this code manually.
7. The quick-fix will delete any comments that happen to be in the original expression. We usually try to preserve user comments when doing any replacements in quick-fixes, using the CommentTracker utility. It looks like the LLM is not aware of it, despite its consistent use in the IntelliJ IDEA Java plugin since at least 2018.
8. For the quick-fix, it uses the old API instead of the shiny, new ModCommands. Of course, the LLM cannot know about ModCommands, as this is an experimental API that appeared very recently, is still heavily changing, and is not documented yet. Luckily the conversion is trivial for simple quick-fixes.

In conclusion, I need to rewrite the solution from the LLM quite significantly to fix these issues. Here’s a part of the diff (the left pane is AI-generated code).

I also need to touch up the documentation, as the style from the LLM is different from our usual style. Nevertheless, at 16:45 I committed the changes, reaching a satisfactory level of quality. The whole task took about 35 minutes, including creating a YouTrack issue and taking the screenshots for this blogpost. Not bad!

Conclusion

I’m not sure how much time was saved by using AI Assistant for this task. I definitely saved some time here, probably 10–20 minutes. It looks like the actual coding doesn’t take too much time. 

More time was needed to find the proper locations for new files, find what to copy-paste, and copy-paste it. I also spent some time checking whether our OrderedBinaryExpression class could help make the inspection robust to argument swapping, but I concluded that it’s simpler not to use it. Finally, I hadn’t initially noticed the problem with <<<, so it took some time to fix that. If you’d like to have a look at the code, it’s on GitHub.

The main problem I see for these kinds of tasks is that AI cannot learn (yet). If this were code produced by a new teammate (assuming there’s no <<< bug), it would be a solid first try. However, I can explain to my teammates that they should use CommentTracker, care about parentheses, and so on during a code review. And the next time, the teammate would do so from the first try, working more efficiently. When AI is able to accumulate and use project-specific knowledge, it will become way more useful.

If you take the time and effort to review AI Assistant-generated code, it may help in speeding up your coding task. Have you tried AI Assistant in IntelliJ IDEA? Let us know how it works for you!

Subscribe to IntelliJ IDEA Blog updates

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!